Machine learning program, machine learning method, machine learning device, determination program, determination method and determination device

Abstract

To accurately determine quality deterioration of raw fish non-destructively.SOLUTION: A computer is allowed to execute a process for acquiring first information based on brightness of each pixel in a first region corresponding to fish meat around middle bone of the fish specified in a plurality of nondestructive inspection images to the fish, and second information based on brightness of each pixel in a second region corresponding to fish meat closer to a body surface of the fish than the first region, and for performing training of machine learning model by using training data in which the first and second information and an evaluation result of the quality deterioration of the fish meat are associated with each other.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a machine learning program, a machine learning method, a machine learning device, a determination program, a determination method, and a determination device. [Background technology]

[0002] One of the indicators that indicates the quality of fish such as tuna is burnt flesh. The cause of burnt flesh has not been fully elucidated, but it is known that one factor is the behavior of the fish when it is reeled in. For example, when a tuna struggles when it is reeled in, its body temperature rises rapidly, causing proteins to condense, resulting in burnt flesh. Temperature conditions are also thought to be an important factor in the occurrence of burnt flesh.

[0003] Fish with discoloration have a whitish appearance and cracks, resulting in a decline in quality. However, discoloration is difficult to detect through visual inspection. Discoloration can only be seen with the naked eye after the fish has been butchered, which can lead to problems such as trouble in trade and a loss of credibility for fishing ports. Discoloration is particularly problematic for tuna, which is an expensive fish.

[0004] It has been found that medical ultrasound equipment can be used to inspect for burns. For example, a technique has been proposed in which an ultrasound inspection device is used as a specimen to obtain ultrasound tomographic images and ultrasound tissue characteristic values ​​such as bulk modulus, acoustic impedance, attenuation constant, and Doppler shift frequency, and the quality of fish is evaluated based on these data. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-155692 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, identifying discoloration from ultrasound tomographic images requires specialized image reading skills, and it is difficult for people without specialized image reading skills to quickly and non-destructively determine the quality deterioration of raw fish using ultrasound tomographic images. In addition, since it is difficult to identify the location of discoloration in a specimen, it is difficult to obtain an ultrasound image that includes the location of discoloration, and in this respect, it is also difficult to quickly and non-destructively determine the quality deterioration of raw fish.

[0007] The disclosed technology has been made in consideration of the above, and aims to determine quality deterioration of raw fish in a non-destructive and accurate manner. [Means for solving the problem]

[0008] In one aspect disclosed in the present application, a computer is caused to execute a process of obtaining first information based on the brightness of each pixel in a first region corresponding to fish meat around a backbone of a fish identified from a plurality of non-destructive inspection images of the fish, and second information based on the brightness of each pixel in a second region corresponding to fish meat closer to the surface of the fish than the first region, and training a machine learning model using training data that associates the first information, the second information, and an evaluation result regarding quality deterioration of the fish meat for each of the plurality of non-destructive inspection images. Effect of the Invention

[0009] According to one aspect, the present invention can determine quality deterioration of fish in a non-destructive and accurate manner. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a quality deterioration determination system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram of the determination device. [Diagram 3] FIG. 3 is a diagram for explaining the measurement of changes in the skeletal reflex. [Figure 4] FIG. 4 is a diagram showing an example of a determination result image. [Diagram 5] FIG. 5 is a diagram showing an outline of a discoloration determination process in an inference phase performed by the determination device according to the embodiment. [Figure 6] FIG. 6 is a flowchart of the overall operation including the training phase and the inference phase of the determination device according to the embodiment. [Figure 7] FIG. 7 is a flowchart of the training process performed in the training phase. [Figure 8] FIG. 8 is a flowchart of the inference process performed in the inference phase. [Figure 9] FIG. 9 is a diagram illustrating an example of a hardware configuration of the determination device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, examples of the machine learning program, the machine learning method, the machine learning device, the determination program, the determination method, and the determination device disclosed in the present application will be described in detail with reference to the drawings. Note that the machine learning program, the machine learning method, the machine learning device, the determination program, the determination method, and the determination device disclosed in the present application are not limited to the following examples. EXAMPLES

[0012] FIG. 1 is a schematic diagram of a quality deterioration determination system according to an embodiment. The quality deterioration determination system 1 includes a determination device 10, an ultrasonic inspection device 20, and an ultrasonic probe 30. The quality deterioration determination system 1 determines whether or not unfrozen raw fish is burnt. The object to be determined may be, for example, a large fish such as tuna, or a medium-sized or small fish. In this embodiment, the quality deterioration determination system 1 determines whether or not the object, raw tuna F, is burnt. The determination device 10 is connected to the ultrasonic inspection device 20.

[0013] The ultrasonic probe 30 is a linear array probe in which an ultrasonic wave emitting section and a reflected wave receiving section are arranged in the same place. The ultrasonic probe 30 receives ultrasonic wave emission setting information such as frequency, power, and direction from the ultrasonic inspection device 20. The ultrasonic probe 30 then irradiates ultrasonic waves to the raw tuna F in accordance with the specified emission setting, and scans the object using an echo method in which the ultrasonic waves reflected from the raw tuna F are received again.

[0014] The ultrasonic inspection device 20 receives an instruction to send ultrasonic waves from the determination device 10. Then, the ultrasonic inspection device 20 transmits ultrasonic irradiation setting information such as frequency, power, and direction to the ultrasonic probe 30, and causes the ultrasonic probe 30 to irradiate ultrasonic waves to the raw tuna F, and the ultrasonic probe 30 acquires a reflected signal of the ultrasonic waves propagated through the raw tuna F. Here, the quality deterioration determination system 1 according to the present embodiment can use medical ultrasonic waves, and outputs ultrasonic waves of, for example, 1 MHz to 20 MHz. The ultrasonic inspection device 20 can continuously perform scanning with ultrasonic waves, and collects data of reflected waves of ultrasonic waves obtained by the ultrasonic probe 30 to obtain a set of data of reflected waves from which an ultrasonic image video can be generated. After that, the ultrasonic inspection device 20 transmits the set of data of reflected waves obtained by scanning the raw tuna F to the image acquisition unit 13 of the determination device 10.

[0015] The judgment device 10 has a judgment image extraction model, a discoloration feature acquisition model, and a discoloration probability calculation model, each of which is a machine learning model. The judgment image extraction model is a machine learning model that extracts a judgment image that displays a usable cross section that can be used to judge the presence or absence of discoloration from an ultrasound image video. The judgment image extraction model includes an image recognition AI (Artificial Intelligence). The discoloration feature acquisition model is a machine learning model that acquires discoloration features in an ultrasound image, which are indicators that can determine the occurrence of discoloration, from the ultrasound image. The discoloration feature acquisition model is an object detection AI. The discoloration probability calculation model is a machine learning model that calculates a discoloration probability, which is the probability that discoloration occurs in an object, from the state of the discoloration features in the ultrasound image. The discoloration probability calculation model is a judgment AI.

[0016] As a process of the training phase, the determination device 10 performs training of the determination image extraction model, the discoloration feature acquisition model, and the discoloration probability calculation model in advance. As a result, the determination device 10 generates a trained determination image extraction model, a trained discoloration feature acquisition model, and a trained discoloration probability calculation model.

[0017] The determination device 10 also performs the following processing as the processing of the inference phase. The determination device 10 transmits an instruction to transmit ultrasound to the ultrasonic inspection device 20, and receives from the ultrasonic inspection device 20 a set of data of reflected waves capable of generating an ultrasonic image video obtained by scanning the raw tuna F. The determination device 10 then generates an ultrasonic image video at a predetermined frame rate from the set of acquired reflected wave data, and calculates the discoloration probability of the object using a determination image extraction model, a discoloration feature acquisition model, and a discoloration probability calculation model. The determination device 10 then displays the calculated discoloration probability of the object on the input / output unit 11 to notify the user. The user is, for example, a buyer of raw tuna.

[0018] Fig. 2 is a block diagram of the determination device. Details of the determination device 10 will be described below with reference to Fig. 2. As shown in Fig. 2, the determination device 10 has an input / output unit 11, an image storage unit 12, an image acquisition unit 13, a discoloration determination unit 14, a learning unit 15, and a model storage unit 16.

[0019] The input / output unit 11 has an output device such as a monitor and an input device such as a keyboard and a mouse. For example, the input / output unit 11 stores a data set of ultrasound images input by an administrator of the determination device 10 using the input device in the image storage unit 12. The input / output unit 11 also adds data to the ultrasound images stored in the image storage unit 12 according to an instruction from a user to add data. The input / output unit 11 also receives an instruction from the discoloration determination unit 14 and causes the output device to display a specified screen.

[0020] The image storage unit 12 is a storage device that stores learning data and ultrasound images. The image storage unit 12 stores a data set of ultrasound images used for learning input from the input / output unit 11. This data set of ultrasound images is an example of "plurality of non-destructive inspection images." The image storage unit 12 also stores a video of ultrasound images of raw tuna F, which is the object of burnt judgment, generated by the image acquisition unit 13.

[0021] The model storage unit 16 holds a judgment image extraction model, a discoloration feature acquisition model, and a discoloration probability calculation model. Here, in this embodiment, the case where three machine learning models are used for discoloration judgment has been described, but some or all of these three may be combined into one machine learning model.

[0022] The administrator of the assessment device 10 checks each ultrasound image included in the data set stored in the image storage unit 12. Then, the administrator designates an assessment image that displays a usable surface that can be used to assess the occurrence of scorching from among the ultrasound images included in the data set stored in the image storage unit 12. It is preferable that the usable surface is an image that shows the backbone and fish scales as a whole and clearly shows the reflection of ultrasound.

[0023] Furthermore, when a discoloration feature that is a feature that makes the occurrence of discoloration suspected is found in the ultrasound image, the administrator adds an annotation indicating the type of discoloration feature to the location indicating the discoloration feature by using the input / output unit 11. In this embodiment, as the annotation, a frame surrounding the area indicating the discoloration feature is added to the ultrasound image.

[0024] In this embodiment, the following are used as the burnt features: a change in the backbone reflection, which is a reflection from the backbone, an echo reflection caused by a change in the protein around the backbone, a strong reaction of the fish scales due to the peeling of the muscle, a change in brightness due to the deterioration of the bloody meat, and the disappearance of the reflection of the fish scales due to the stiffening of the protein. Here, in this embodiment, the above-mentioned five burnt features are used, but other information obtained from an ultrasonic image that can be used to determine burnt may be used as the burnt features.

[0025] The administrator adds annotations to the areas of each ultrasound image that show the burnt features for the other four burnt features that appear directly in the ultrasound image other than the change in the backbone reflection. For example, if a brightened area is observed in a part of a fish scale shown in an ultrasound image, the administrator adds an annotation to the area surrounding that area indicating a strong reaction of the fish scale due to muscle detachment. If a brightened area is observed in the bloody meat, the administrator adds an annotation to the area surrounding that area indicating a change in brightness due to deterioration of the bloody meat. If a partial or complete disappearance of a fish scale is observed, the administrator adds an annotation to the area surrounding that area indicating the disappearance of the fish scale reflection due to protein stiffening.

[0026] In addition, for the characteristics of the change in the backbone reflection of the ultrasound, the administrator adds an annotation for the backbone reflection as shown in the ultrasound image 101 of Fig. 3. Fig. 3 is a diagram for explaining the measurement of the change in the backbone reflection. The ultrasound image 101 of Fig. 3 is an image for judgment, and includes images of the surface 102 of the fish, the fish scales 103, and the backbone 104.

[0027] Here, the reflection from the backbone 104 is affected by the physical properties of the fish meat from the surface 102 to the backbone 104. For example, Young's modulus and viscosity coefficient are examples of physical properties that affect sound propagation. When burnt meat occurs, the physical properties of the fish meat change, so it is important to quantify the backbone reflection, which is the reflection from the backbone 104, in order to determine whether burnt meat has occurred. Therefore, the determination device 10 according to this embodiment quantifies the magnitude of the backbone reflection by the following procedure.

[0028] The backbone vicinity region 111 has a fixed shape including the backbone 104. This fixed shape is, for example, a region including the backbone 104 and having a width of 1 to 2 cm in the direction connecting the backbone 104 and the surface 102. This backbone vicinity region 111 is an example of a "first region." Moreover, the surface vicinity region 112 is a region of the same shape as the backbone vicinity region 111, including a region 1 cm deep from the body surface of the tuna. This surface vicinity region 112 is an example of a "second region."

[0029] If no discoloration occurs and there is no change in the physical properties, the ultrasonic waves are reflected by the backbone 104, so the amplitude and brightness of the reflected waves shown in the backbone vicinity region 111 become large. In contrast, if discoloration occurs, the ultrasonic waves undergo diffuse reflection in the discolored region before reaching the backbone 104 due to the change in the physical properties, and less ultrasonic waves are reflected by the backbone 104, so the amplitude and brightness of the reflected waves shown in the backbone vicinity region 111 decrease. On the other hand, the reflection of ultrasonic waves in the surface vicinity region 112 is not easily affected by discoloration, so the amplitude and brightness remain constant regardless of whether discoloration occurs or not.

[0030] Therefore, the determination device 10 according to the present embodiment uses the ratio of the average luminance of each pixel included in the backbone vicinity region 111 to the average luminance of each pixel included in the surface vicinity region 112 as an index of backbone reflex. Specifically, the determination device 10 uses the value obtained by dividing the average luminance of each point included in the backbone vicinity region 111 by the average luminance of each point included in the surface vicinity region 112 as an index of backbone reflex. That is, the smaller the value of this index, the higher the probability that scorching has occurred. Conversely, the larger the value of this index, the higher the probability that scorching has not occurred. This average luminance of each pixel included in the backbone vicinity region 111 is an example of the "first information". Also, the average luminance of each pixel included in the surface vicinity region 112 is an example of the "second information".

[0031] For the above reasons, the administrator uses the input / output unit 11 to add, for example, the midbone vicinity region 111 and the surface vicinity region 112 to the ultrasound image 101 as annotations for changes in midbone reflection. The administrator also uses the input / output unit 11 to add, for each ultrasound image included in the data set stored in the image storage unit 12, annotations for changes in midbone reflection.

[0032] The administrator also uses input / output unit 11 to assign to each ultrasound image information regarding whether or not the raw tuna from which each ultrasound image was taken was discolored, which is included in the data set stored in image storage unit 12.

[0033] The learning unit 15 acquires a data set of ultrasound images from the image storage unit 12. The learning unit 15 then trains, i.e., learns, the determination image extraction model using each ultrasound image included in the data set and information on whether each ultrasound image is an image for determination. The trained determination learning extraction model is capable of receiving an input of an ultrasound image and outputting an image for determination extracted from the ultrasound image.

[0034] The learning unit 15 also trains a discoloration feature acquisition model using information on discoloration features indicated by annotations added to each of the ultrasound images included in the data set. The trained discoloration feature acquisition model receives an input of an ultrasound image, and when a discoloration feature is present in the ultrasound image, acquires information on the position of the discoloration feature together with information on the type of the discoloration feature.

[0035] The learning unit 15 also trains the burn probability calculation model using the information on the position and type of the burnt feature extracted from the ultrasound image, and the information on the presence or absence of burnt of the raw tuna from which the ultrasound image from which the burnt feature was extracted was taken. This information on the presence or absence of burnt is an example of an "evaluation result."

[0036] In addition, since there are types of discoloration features that have a strong influence on the judgment of discoloration occurrence and those that have a secondary influence, the weight of each type is adjusted in the learning of the discoloration probability calculation model. However, if the discoloration probability calculation model is a simple model, the discoloration probability calculation model may equalize the weight of each type of discoloration feature and simply use the sum of the discoloration probabilities of each type as the result of discoloration judgment. The trained discoloration probability calculation model receives input of an ultrasound image, discoloration features extracted from the ultrasound image, and information on the type of discoloration, and calculates the discoloration probability of the raw tuna whose ultrasound image was taken.

[0037] For example, when learning the probability of discoloration according to the backbone reflex, the learning unit 15 performs the following process. The learning unit 15 inputs training data in which the average luminance value of each pixel included in the backbone vicinity region 111, the average luminance value of each pixel included in the surface vicinity region 112, and information on the presence or absence of discoloration are associated with each other to the discoloration probability calculation model. The discoloration probability calculation model performs learning based on the association between the value of the index of the backbone reflex, which is obtained by dividing the average luminance value of each pixel included in the backbone vicinity region 111 by the average luminance value of each pixel included in the surface vicinity region 112, and the information on the presence or absence of discoloration.

[0038] Furthermore, when learning the probability of discoloration according to echo reflection caused by changes in proteins around the backbone, the learning unit 15 performs the following process. The learning unit 15 acquires images of reflected ultrasound from the part around the backbone identified from each ultrasound image included in the data set, and inputs the images as training data to the discoloration probability calculation model in association with information on the presence or absence of discoloration. The discoloration probability calculation model learns based on the association between the images of reflected ultrasound from the part around the backbone and information on the presence or absence of discoloration. This image of reflected ultrasound from the part around the backbone identified from each ultrasound image included in the data set is an example of "third information."

[0039] Furthermore, when learning the probability of discoloration according to a strong reaction of the fish scales due to muscle detachment or the disappearance of the reflection of the fish scales due to protein stiffening, the learning unit 15 performs the following process. The learning unit 15 acquires images of the parts corresponding to the fish scales identified from each ultrasound image included in the data set, associates them with information on the presence or absence of discoloration, and inputs them to the discoloration probability calculation model as training data. The discoloration probability calculation model learns based on the correspondence between the images of the parts corresponding to the fish scales and the information on the presence or absence of discoloration. These images of the parts corresponding to the fish scales identified from each ultrasound image included in the data set are an example of "fourth information."

[0040] Furthermore, when learning the probability of discoloration according to the change in brightness due to the deterioration of the bloody meat, the learning unit 15 performs the following process. The learning unit 15 acquires images of the parts corresponding to the bloody meat identified from each ultrasound image included in the data set, associates them with information on the presence or absence of discoloration, and inputs them to the discoloration probability calculation model as training data. The discoloration probability calculation model learns based on the correspondence between the images of the parts corresponding to the bloody meat and the information on the presence or absence of discoloration. These images of the parts corresponding to the bloody meat identified from each ultrasound image included in the data set are an example of "fifth information."

[0041] Here, in the above explanation, the learning regarding the discoloration features has been explained separately, but the learning unit 15 can cause the discoloration probability calculation model to learn each discoloration feature collectively. By learning the discoloration features collectively, the learning unit 15 can determine the weight of each discoloration feature and cause the discoloration probability calculation model to learn so as to calculate the discoloration probability of the target raw tuna F from each discoloration feature to which the weight has been added.

[0042] The learning unit 15 stores the trained determination image extraction model, the trained discoloration feature acquisition model, and the trained discoloration probability calculation model in the model storage unit 16. With the above, the pre-training phase of the determination device 10 is completed.

[0043] The image acquisition unit 13 receives an input of a set of reflected wave data obtained by scanning the raw tuna F from the ultrasonic inspection device 20. Here, the raw tuna F is scanned, for example, by the user P moving the ultrasonic probe 30 between the vicinity of the ventral fin and the vicinity of the second dorsal fin. At this time, the user P performs scanning by positioning the ultrasonic probe 30 so that ultrasonic waves are irradiated in the direction of the backbone of the raw tuna F. The image acquisition unit 13 generates an ultrasonic image video using the acquired reflected wave data and stores it in the image storage unit 12.

[0044] The burnt determination unit 14 acquires an ultrasound image video obtained by scanning the raw tuna F from the image storage unit 12. Next, the burnt determination unit 14 inputs an ultrasound image for each frame obtained from the ultrasound image video to a trained image extraction model for judgment, and acquires an image for judgment in which a usable cross section showing the backbone and fish scales as a whole is displayed and output from the image extraction model for judgment. Here, the image for judgment output by the image extraction model for judgment can be specified in advance as, for example, 3 frames, 5 frames, etc.

[0045] Next, the discoloration judgment unit 14 inputs the extracted judgment image into a trained discoloration feature acquisition model, and acquires information on the position of the discoloration feature portion in the judgment image output from the discoloration feature acquisition model and information on the type of the discoloration feature.

[0046] Next, the discoloration determination unit 14 inputs information on the position of the discoloration feature portion and information on the type of the discoloration feature into the discoloration probability calculation model, and obtains a discoloration determination result including the discoloration probability of the raw tuna F output from the discoloration probability calculation model. Here, the discoloration determination result may include information indicating the discoloration probability for each type of discoloration feature and the location of the discoloration feature that has occurred.

[0047] For example, the discoloration probability calculation model estimates the occurrence of discoloration due to the backbone reflex as follows: The discoloration probability calculation model identifies the backbone from the judgment image using object detection AI. The discoloration probability calculation model then sets the backbone vicinity region and the surface vicinity region based on the identified backbone and the surface. The discoloration probability calculation model then divides the average brightness of each point in the backbone vicinity region by the average brightness of each point in the surface vicinity region to calculate the value of the backbone reflex index. The discoloration probability calculation model then calculates the probability of discoloration due to the backbone reflex using the calculated backbone reflex index.

[0048] The discoloration probability calculation model calculates the discoloration probability of other discoloration features from the image. Then, the discoloration probability calculation model weights each discoloration feature with its respective weight, and then combines all the discoloration probabilities to calculate the overall discoloration probability of raw tuna F.

[0049] Thereafter, the burnt determination unit 14 generates a determination result image showing the burnt probability of the raw tuna F. FIG. 4 is a diagram showing an example of the determination result image. For example, the burnt determination unit 14 can generate a determination result image 200 shown in FIG. 4. The determination result image 200 displays an example ultrasound image 201, burnt feature information 221 and 222, and a burnt probability 223. The burnt determination unit 14 may use any of the determination images used for the burnt determination as the ultrasound image 201, or may generate a new image according to the determination result based on the determination image or the like. In addition, the burnt determination unit 123 may add a mark that highlights the burnt feature area in the ultrasound image 201, and generate the burnt feature information 221 and 222 as information according to the mark. With the above, the inference phase by the determination device 10 is completed.

[0050] Then, the discoloration judgment unit 14 transmits the generated judgment result screen to the input / output unit 11 and displays it on the display device, thereby notifying the user P of the judgment result for the raw tuna F. By checking the judgment result, the user P can confirm the probability of discoloration occurring on the raw tuna F, and can also confirm the condition of the raw tuna F by the image on which the judgment was based.

[0051] 5 is a diagram showing an overview of the discoloration determination process in the inference phase by the determination device 10 according to the embodiment. Next, the discoloration determination process in the inference phase by the determination device 10 according to the embodiment will be summarized again with reference to FIG.

[0052] The user P moves the ultrasonic probe 30 so as to cross the area W between the vicinity of the ventral fin and the vicinity of the second dorsal fin of the raw tuna F. Then, the determination device 10 acquires data of the reflected waves obtained by scanning the ultrasonic probe 30, and obtains an ultrasonic image video 150 obtained by scanning the raw tuna F (step S1).

[0053] Next, the determination device 10 inputs the ultrasound images of each frame of the ultrasound image video 150 to a determination-use image extraction model, and extracts determination-use images 151 to 153 as the output (step S2).

[0054] Next, the determination device 10 inputs the determination images 151-153 into a discoloration feature acquisition model, and acquires position information 161-163 of the discoloration feature portions and the types of the discoloration features as the output (step S3). Here, the determination images 151-153 include various images such as images without discoloration features and images with parts of the detected discoloration features.

[0055] Next, the judgment device 10 inputs the judgment images 151-153 and the information on the discoloration features into a discoloration probability calculation model, and calculates the discoloration probability 170 of the raw tuna F as the output (step S4). After that, the judgment device 10 notifies the user P of the judgment result of the discoloration judgment, including the calculated discoloration probability 170.

[0056] 6 is a flowchart of the entire operation including the training phase and the inference phase of the determination device according to the embodiment. Next, the flow of the entire operation including the learning phase and the inference phase of the determination device 10 according to the embodiment will be described with reference to FIG.

[0057] The determination device 10 executes a determination image extraction model, a discoloration probability calculation model, and a training process for the discoloration probability calculation model, which are used to infer the presence or absence of discoloration (step S11).

[0058] After the training is completed, the judgment device 10 performs an inference process to determine whether or not discoloration has occurred on the target fish using the learned judgment image extraction model, the discoloration probability calculation model, and the discoloration probability calculation model, and calculates the discoloration probability of the target fish (step S12).

[0059] Thereafter, the determination device 10 notifies the user of the determination result including the calculated probability of discoloration (step S13).

[0060] Fig. 7 is a flowchart of a training process performed in the training phase. The process shown in Fig. 7 corresponds to an example of the training process executed in step S1 in Fig. 6. Next, a flow of the process in the training phase by the determination device 10 according to the embodiment will be described with reference to Fig. 7.

[0061] The image storage unit 12 receives an input from the input / output unit 11, for example, and acquires and stores a data set of an ultrasound image (step S111).

[0062] The user annotates the burnt features of each ultrasound image included in the data set held by the image storage unit 12 (step S112). At this time, information on the type of the detected burnt features is also added to the ultrasound image in which the burnt features are present. In addition, information on whether or not the photographed fish has burnt and information on whether or not the image contains a usable cross section are also added to each ultrasound image.

[0063] Next, the learning unit 15 uses the ultrasound image dataset and information on whether each ultrasound image includes a cross section or not to cause the judgment image extraction model stored in the model storage unit 16 to learn judgment images that display usable cross sections (step S113).

[0064] Next, the learning unit 15 uses the data set of the ultrasound images and the information on the positions and types of the discoloration features to cause the discoloration feature acquisition model stored in the model storage unit 16 to learn the discoloration features in the ultrasound images (step S114).

[0065] Next, the learning unit 15 uses the ultrasound image data set, information on the position and type of discoloration features, and information on whether discoloration has occurred to train the discoloration probability calculation model stored in the model storage unit 16 to calculate the discoloration probability from the discoloration features (step S115).

[0066] Fig. 8 is a flowchart of an inference process performed in the inference phase. The process shown in Fig. 8 corresponds to an example of the inference process executed in step S2 in Fig. 6. Next, the flow of the process in the inference phase by the quality deterioration determination system 1 according to the embodiment will be described with reference to Fig. 8.

[0067] The user P brings the ultrasonic probe 30 into contact with the raw tuna F between the vicinity of the ventral fin and the vicinity of the second dorsal fin and moves it between those two points. The ultrasonic probe 30 receives ultrasonic irradiation setting information such as frequency, power and direction from the ultrasonic inspection device 20, and scans the raw tuna F by irradiating the raw tuna F with ultrasonic waves according to the settings (step S121).

[0068] The ultrasonic inspection device 20 acquires data on reflected waves of ultrasonic waves propagated through the raw tuna F using the ultrasonic probe 30. The image acquisition unit 13 acquires data on reflected waves of ultrasonic waves obtained by scanning the raw tuna F. Then, the image acquisition unit 13 acquires an ultrasonic image video from the acquired reflected wave data and stores it in the image storage unit 12 (step S122).

[0069] Next, the discoloration determiner 14 inputs the ultrasonic image moving image to a determination use image extraction model, and extracts a determination use image as the output (step S123).

[0070] Next, the discoloration determination unit 14 inputs the extracted determination image to a discoloration feature acquisition model, and acquires, as an output, information on the position of the discoloration feature portion and the type of the discoloration feature, thereby acquiring discoloration feature information (step S124).

[0071] Next, the discoloration determination unit 14 inputs the determination image and the discoloration feature information to a discoloration probability calculation model, and calculates the discoloration probability 170 of the raw tuna F as the output (step S125).

[0072] Thereafter, the determination device 10 outputs the result of the discoloration determination including the calculated discoloration probability, and notifies the user P of the result (step S126).

[0073] In this embodiment, the ultrasonic image is used to judge the discoloration, but the image obtained by other techniques may be used as long as it is a non-destructive inspection image that can obtain the internal state of the test object non-destructively and show the discoloration characteristics. For example, an X-ray image or a CT image may be used as long as it is an image that can judge the discoloration characteristics.

[0074] As described above, the determination device according to the present embodiment automatically calculates the probability of burnt fish using ultrasonic images obtained by scanning raw fish and notifies the user. This allows the user to determine whether or not the fish is burnt without dissecting the fish, and allows the burnt fish to be determined non-destructively and with high accuracy. This allows the buying and selling of fish to be done with peace of mind, and makes it possible to promote and activate the distribution of fish.

[0075] In addition, the determination device according to this embodiment determines whether or not discoloration has occurred from the reflection of the backbone. The backbone does not change and exists at a specific position, and its position is easy to specify. By determining the occurrence of discoloration using the backbone, the accuracy of detection of discoloration can be improved. For example, if light discoloration has occurred over a wide area, it is difficult to specify the discoloration from the image. In addition, it is difficult to specify where the discoloration has occurred, and it is difficult to obtain an image targeting the discoloration. In contrast, since the position of the backbone is fixed, it is easy to obtain the backbone reflection, and if there is discoloration between the surface and the backbone, the brightness of the backbone reflection will be greatly reduced even if the discoloration is light and widespread. Therefore, the determination device according to this embodiment can reliably detect the occurrence of discoloration.

[0076] (Hardware) Next, a hardware configuration example of the determination device 10 will be described. Fig. 9 is a diagram showing an example of the hardware configuration of the determination device. As shown in Fig. 9, the determination device 10 has a processor 91, a memory 92, a hard disk 93, an input / output device 94, and a communication device 95. The processor 91 is connected to the memory 92, the hard disk 93, the input / output device 94, and the communication device 95 via a bus.

[0077] The communication device 95 is a communication interface with an external device. For example, the communication device 95 relays communication between the processor 91 and the ultrasonic inspection device 20.

[0078] The input / output device 94 includes a monitor, a keyboard, a mouse, etc. The input / output device 94 realizes the function of the input / output unit 11 illustrated in FIG.

[0079] The hard disk 93 is an auxiliary storage device. The hard disk 93 realizes the functions of the image storage unit 12 and the model storage unit 16 illustrated in Fig. 2. The hard disk 93 also stores various programs including programs for realizing the functions of the image acquisition unit 13, the discoloration determination unit 14, and the learning unit 15 illustrated in Fig. 2.

[0080] The processor 91 reads out various programs stored in the hard disk 93, expands them in the memory 92, and executes them. In this way, the processor 91 realizes the functions of the image acquisition unit 13, the discoloration determination unit 14, and the learning unit 15 illustrated in FIG.

[0081] The determination device 10 can also realize the same functions as those of the above-described embodiment by reading the above-described program from a recording medium using a medium reading device and executing the read program. Note that the program referred to here is not limited to being executed by a specific determination device 10. For example, the present invention can be similarly applied to cases where another computer or server executes a program, or where these execute a program in cooperation with each other.

[0082] This program can be distributed via a network such as the Internet. This program can also be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer. [Explanation of symbols]

[0083] 1. Quality Deterioration Judgment System 10 Judgment device 20 Ultrasonic inspection equipment 30 Ultrasound Probe 11 Input / output section 12 Image storage section 13 Image acquisition section 14. Discoloration Judgment Section 15 Learning Department 16 Model Storage Section

Claims

1. obtaining first information based on the brightness of each pixel in a first region corresponding to fish meat around a backbone of the fish identified from a plurality of non-destructive inspection images of the fish, and second information based on the brightness of each pixel in a second region corresponding to fish meat closer to a body surface of the fish than the first region; A machine learning model is trained using training data in which the first information, the second information, and an evaluation result regarding quality deterioration of the meat of the fish are associated with each other for each of the plurality of non-destructive inspection images. A machine learning program that causes a computer to execute processing.

2. obtaining first information based on the brightness of each pixel in a first region corresponding to fish meat around a backbone of the fish identified from a plurality of non-destructive inspection images of the fish, and second information based on the brightness of each pixel in a second region corresponding to fish meat closer to a body surface of the fish than the first region; A machine learning model is trained using training data that associates the first information, the second information, and an evaluation result regarding quality deterioration of the meat of the fish for each of the plurality of non-destructive inspection images, and specific first information in the first region and specific second information in the second region specified in the non-destructive inspection image of the fish to be inspected are input to the machine learning model, and quality deterioration of the fish to be inspected is judged based on an output obtained by inputting the specific first information in the first region and the specific second information in the second region specified in the non-destructive inspection image of the fish to be inspected. A determination program that causes a computer to execute a process.

3. the process of acquiring the first information and the second information includes a process of acquiring an average value of luminance of each pixel in the first region as the first information, and a process of acquiring an average value of luminance of each pixel in the second region as the second information; The training process includes a process of training the machine learning model using a ratio between the first information and the second information and a correspondence between the evaluation result and the ratio.

2. The machine learning program according to claim 1 .

4. obtaining third information based on an image of a portion around a backbone identified from the plurality of non-destructive inspection images; The machine learning model is trained using training data in which the third information and the evaluation result are associated with each other. The machine learning program according to claim 1 , further comprising causing the computer to execute a process.

5. Fourth information based on an image of a part corresponding to fish scales identified from the plurality of non-destructive inspection images is obtained, and the machine learning model is trained using training data in which the fourth information and the evaluation result are associated with each other. The machine learning program according to claim 1, characterized in that the program causes the computer to execute processing.

6. Fifth information based on an image of a portion corresponding to bloody meat identified from the plurality of non-destructive inspection images is obtained, and the machine learning model is trained using training data in which the fifth information is associated with the evaluation result. The machine learning program according to claim 1, characterized in that the program causes the computer to execute processing.

7. The computer obtaining first information based on the brightness of each pixel in a first region corresponding to fish meat around a backbone of the fish identified from a plurality of non-destructive inspection images of the fish, and second information based on the brightness of each pixel in a second region corresponding to fish meat closer to a body surface of the fish than the first region; A machine learning model is trained using training data in which the first information, the second information, and an evaluation result regarding quality deterioration of the meat of the fish are associated with each other for each of the plurality of non-destructive inspection images. A machine learning method comprising:

8. The computer obtaining first information based on the brightness of each pixel in a first region corresponding to fish meat around a backbone of the fish identified from a plurality of non-destructive inspection images of the fish, and second information based on the brightness of each pixel in a second region corresponding to fish meat closer to a body surface of the fish than the first region; A machine learning model is trained using training data that associates the first information, the second information, and an evaluation result regarding quality deterioration of the meat of the fish for each of the plurality of non-destructive inspection images, and specific first information in the first region and specific second information in the second region specified in the non-destructive inspection image of the fish to be inspected are input to the machine learning model, and quality deterioration of the fish to be inspected is judged based on an output obtained by inputting the specific first information in the first region and the specific second information in the second region specified in the non-destructive inspection image of the fish to be inspected. A determination method comprising:

9. an acquisition unit that acquires first information based on the luminance of each pixel in a first region corresponding to fish meat around a backbone of the fish identified from a plurality of non-destructive inspection images of the fish, and second information based on the luminance of each pixel in a second region corresponding to fish meat closer to a body surface of the fish than the first region; a learning unit that trains a machine learning model using training data in which the first information, the second information, and an evaluation result regarding quality deterioration of the fish meat are associated with each other for each of the plurality of non-destructive inspection images; A machine learning device comprising:

10. an acquisition unit that acquires first information based on the luminance of each pixel in a first region corresponding to fish meat around a backbone of the fish identified from a plurality of non-destructive inspection images of the fish, and second information based on the luminance of each pixel in a second region corresponding to fish meat closer to a body surface of the fish than the first region; a determination unit that determines quality deterioration of the fish to be inspected based on an output obtained by inputting specific first information in the first region and specific second information in the second region specified in the non-destructive inspection image of the fish to be inspected into a machine learning model trained using training data in which the first information, the second information, and an evaluation result regarding quality deterioration of the meat of the fish are associated with each other for each of the plurality of non-destructive inspection images; A determination device comprising:

Images