Information processing method, storage medium, and information processing apparatus

Abstract

According to the present disclosure, there is provided an information processing method executed by a computer, including: generating, by the computer, a simulation image (SI) as input data (TID) through computer graphics; generating label information according to parameter information used in the simulation; and generating correct data (TOD) including the simulation image (SI) and the label information.

Description

Technical Field

[0001] This invention relates to information processing methods, programs, and devices. Background Technology

[0002] Deep learning has achieved remarkable results in the fields of computer vision and speech recognition. For example, automatically counting animals in livestock farms has become a research topic, and it is hoped that deep learning methods can be used to improve this problem.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2019 / 045091 Summary of the Invention

[0006] The analytical accuracy of deep learning depends on the training data. However, creating accurate training data is not easy. Furthermore, in cases involving subjects like schools of fish, there are many overlapping and invisible individuals, making it difficult to accurately infer the required information using deep learning alone.

[0007] Therefore, one objective of this disclosure is to provide an information processing method, program, and apparatus capable of easily generating high-precision training data. Another objective of this disclosure is to provide an information processing method, program, and apparatus capable of accurately inferring the information contained in actual data.

[0008] According to one aspect of this disclosure, an information processing method is provided, executed by a computer, comprising the following steps: generating a simulated image as input data through computer drawing; generating label information based on parameter information used in the simulation; and generating correct data including the simulated image and the label information. Additionally, according to this disclosure, a program for causing a computer to execute the information processing method and an information processing apparatus for executing the aforementioned information processing method are provided.

[0009] Invention Effects

[0010] According to this disclosure, high-precision training data can be easily generated. Attached Figure Description

[0011] Figure 1 It is a diagram that illustrates an overview of information processing.

[0012] Figure 2 It is a graph that represents changes in image elements.

[0013] Figure 3 It is a graph that represents changes in image elements.

[0014] Figure 4It is a graph that represents changes in image elements.

[0015] Figure 5 This is a diagram illustrating machine learning used to remove noise caused by reflected images.

[0016] Figure 6 This is a diagram illustrating machine learning used to remove noise caused by shadows.

[0017] Figure 7 This is a diagram representing an example of group model 23.

[0018] Figure 8 This is a simulated image of the group when it is not being fed.

[0019] Figure 9 It is a simulated image of the group during feeding.

[0020] Figure 10 These are images of the group when it is not being fed.

[0021] Figure 11 These are images of the group being fed.

[0022] Figure 12 This is a diagram representing an example of an individual model.

[0023] Figure 13 It is a diagram representing the simulation of an individual's actions.

[0024] Figure 14 It is a picture showing sturgeon raised in a fishpond.

[0025] Figure 15 This is a diagram illustrating an example of an information processing device.

[0026] Figure 16 This is a diagram illustrating an example of the process of calculating the number of individuals in a group.

[0027] Figure 17 This is a diagram illustrating an example of individual tracking.

[0028] Figure 18 This diagram illustrates an example of methods for identifying individuals and detecting abnormal behavior.

[0029] Figure 19 This is a flowchart illustrating an example of an information processing method.

[0030] Figure 20 This is a diagram illustrating an example of the hardware structure of an information processing device.

[0031] Explanation of reference numerals in the attached figures

[0032] 1. Information processing device; 21. Analysis model; 23. Group model; 24. Individual model; 25. Abnormal behavior model; 29. ​​Program; CI captured image; CSI corrected image; FLK group; IE image element; OB individual; SI simulated image; TID input data; TOD correct data. Detailed Implementation

[0033] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following embodiments, repeated descriptions are omitted by using the same symbols to mark the same parts.

[0034] 1. Overview of Information Processing

[0035] Figure 1 This is a diagram illustrating an overview of the information processing described in this disclosure.

[0036] This disclosure presents a novel method for generating training data using CG (Computer Graphics) and a Data Driven method for bridging the differences between CG images and actual data (captured images). By generating training data using CG images, it is possible to generate a large amount of high-precision training data. Furthermore, by generating CG images that approximate actual data, necessary information can be accurately inferred from the parameters of the CG. This disclosure will now be described in detail.

[0037] In this disclosure, the target TG is analyzed using image analysis methods based on the captured image CI of the target TG. The information processing method of this disclosure can be applied to a wide range of fields, including aquaculture, agriculture, forestry, and industry. The analysis object is appropriately set according to the technical field and the purpose of the analysis. For example, in the aquaculture field, the number of farmed fish, their growth status, predatory behavior, and abnormal behavior are used as analysis objects. In the agricultural field, the growth status and harvest quantity of crops are used as analysis objects. In the forestry field, the growth status of trees and the amount of timber are used as analysis objects. In the industrial field, the quantity of products and the presence or absence of defective products are used as analysis objects.

[0038] The following examples illustrate how the information processing methods disclosed herein are applied to the tracking, abnormal behavior detection, and fish count of farmed fish.

[0039] 2. Generation of training data

[0040] In this disclosure, computer simulations are used to generate 3D CG images of the swarm FLK and individual OBs. The generated CG images are used as training data for deep learning.

[0041] The training data comprises a large number of data sets. Each data set includes input data TID and correct data TOD. In this disclosure, a simulated image SI, serving as the input data TID, is generated by computer graphics. The correct data TOD is generated by appending label information to the input data TID. The label information represents ground truth. The label information is automatically generated based on the parameter information used in the simulation.

[0042] The simulation object is determined based on the learning objective. For example, when counting the number of farmed fish, a group FLK simulation is performed. When tracking individual farmed fish, an individual OB simulation is performed.

[0043] Figure 1 An example is the generation of simulated images (SIs) of individual fish (OBs) for the purpose of tracking farmed fish. Label information includes classification information for categorizing the individual OB and location information for the individual OB. Classification information includes, for example, the species (fish type, etc.) and morphological characteristics (fin deformities, defects, body color, patterns, etc.) of the individual OB. Location information includes, for example, information indicating the center position of the individual OB and the extent of its presence. Location information is generated, for example, using three-dimensional coordinate information. Figure 1 In the example, the coordinates of the center of individual OB (b x ,b y ,b z And the depth b of the boundary cube containing the minimum size of the individual OB. d Width b w And height b h The relevant information is represented as location information. Based on the location information of an individual OB (indicating the existence of a range), the size information of the individual OB is extracted.

[0044] Correct TOD data can include image elements (IEs) for identifying individual OBs. An image element IE is, for example, a frame element representing a boundary cube. Image elements IEs are automatically generated based on the location information of the individual OBs used in the simulation. Figure 1 In the example, a boundary cube is used to represent the image element IE, but the image element IE is not limited to this. An image element IE may, for example, include a frame element that surrounds an individual OB or a color element used to identify an individual OB using color.

[0045] Figures 2 to 4 This is a graph representing the changes in image elements (IE).

[0046] Figure 2 The example is reproduced Figure 1 The boundary cube shown. Figure 3Examples include replacing individual OBs with outlines or outlining individual OBs with thick lines. Figure 4 An example is using color to distinguish multiple individual OBs. Figure 2 boundary cube and Figure 3 The outline is equivalent to the frame element that surrounds the individual OB. Figure 4 Color differentiation is equivalent to assigning color to elements. When there are multiple individual OBs, the color of a frame element can differ according to the individual OBs, or it can differ according to the classification information of the individual OBs.

[0047] 3. Deep Learning

[0048] return Figure 1 In this disclosure, machine learning (with teacher training) is performed in a DNN (Deep Neural Network) using input data TID and correct data TOD as training data. If the input data RID is fed into the learned DNN, output data ROD is generated by appending label information and image element IE to the input data RID.

[0049] For example, if a captured image CI is input into a DNN, the DNN extracts individual OBs from the captured image CI. The DNN generates classification and location information for each individual OB based on its features. The DNN generates label information for each individual OB, including both classification and location information. The DNN generates image elements IE for each individual OB based on its location information. The DNN generates a corrected image CCI, which is the captured image CI with the image elements IE of each individual OB appended to it. The DNN then appends the label information of each individual OB to the corrected image CCI to generate the output data ROD.

[0050] 4. Noise Removal

[0051] Figure 5 as well as Figure 6 This is a diagram illustrating machine learning used to remove noise.

[0052] To improve the accuracy of image parsing performed by a DNN, machine learning can be used to remove noise contained in the captured image CI. For example, if a camera is placed in a fishpond to capture images, the reflected image MIR of an individual OB moving near the water surface WS and the wall WL is projected onto the water surface WS and the wall WL. The shadow SH of the individual OB is projected onto the bottom of the fishpond. Such reflected images MIR and shadow SH become noise, potentially reducing the accuracy of image parsing. Therefore, in this disclosure, for example, a simulated image SI of the reflected image MIR and shadow SH is generated, and training data using the simulated image SI is generated.

[0053] exist Figure 5 In the example, a simulated image SI, including the individual OB and the reflective image MIR, is generated as input data TID. The correct data TOD is the simulated image SI obtained by removing the reflective image MIR from the input data TID. If this input data TID and the correct data TOD are used as training data, a DNN is obtained that removes the reflective image MIR from the captured image CI. Figure 5 In the example, the input data TID and the correct data TOD contain individual OBs, but these data do not necessarily need to contain individual OBs. For example, even if the training data obtained by removing individual OBs from the input data TID and the correct data TOD mentioned above is used, a DNN with reflective images MIR removed will still be generated.

[0054] exist Figure 6 In the example, a simulated image SI including the individual OB and its shadow SH is generated as input data TID. The correct data TOD is the simulated image SI from the input data TID with the shadow SH removed. If this input data TID and the correct data TOD are used as training data, a DNN is obtained that removes the shadow SH from the captured image CI. Figure 6 In the example, the input data TID and the correct data TOD contain individual OBs, but these data do not necessarily contain individual OBs. For example, even if the training data obtained by removing individual OBs from the input data TID and the correct data TOD mentioned above is used, a DNN with shadow SH removed will still be generated.

[0055] 5. Simulation Model

[0056] Simulated images SI of the swarm FLK and individual OB can be generated using swarm model 23 (see reference). Figure 15 ) and individual model 24 (refer to) Figure 15 (Generation). Group model 23 is a simulation model based on the statistical properties of group FLK. Individual model 24 is a simulation model based on the morphological characteristics of individual OB. The following is a brief description of the simulation models.

[0057] 5-1. Group Model

[0058] Figure 7 This is a diagram representing an example of group model 23.

[0059] Group model 23 is generated, for example, by combining a predator model representing predatory behavior during feeding into a Boids model that specifies three action rules: classification, alignment, and combination. In the predator model, a repulsion region RA is set outside the mixed region CA, where separation action rules apply, to encourage movement in the opposite direction to the density direction. The predator model is generated by simulating how individual OBs obtain the prey presented in their field of vision (Fish Vision Simulation). The predator model can also be used to optimize prey based on the species of individual OBs and the density within the cage.

[0060] In swarm model 23, parameters for swarm FLK include, for example, the alignment degree of individual OBs, swarm formation degree, perspective, fastest speed, bait recognition speed, bait greed, duration of continuous feeding (feeding amount until satiety), time for one bait pecking, movement speed while feeding, size of the mixing region CA, and size of the repulsion region RA. These parameters vary depending on the type and size of the individual OBs.

[0061] Figure 8 SI is a simulated image of the FLK group when it is not being fed. Figure 9 It is a simulated image SI of the FLK group during feeding. Figure 10 Image CI of a group of FLKs taken when not being fed. Figure 11 This is a photograph (CI) of a group of FLKs during feeding.

[0062] like Figure 8 as well as Figure 10 As shown, when not being fed, the FLK group generally follows three action rules: separation, alignment, and aggregation. Therefore, a large number of individual OBs are densely packed within a narrow area. Figure 9 as well as Figure 11 As shown, during feeding, the FLK group behaves according to four action rules: separation, alignment, combination, and repulsion. The repulsion action rule causes the FLK group to spread, thus reducing the density of the FLK group.

[0063] like Figure 10 As shown, in dense clusters like FLK, there are many overlapping and invisible individuals (OBs), making it difficult to accurately count the number of individuals. Therefore, according to... Figure 8 The simulated image SI is shown to estimate the number of individuals. The shape and size of the FLK group are changed by adjusting the simulation parameters. Therefore, by adjusting the parameter information until the simulated image SI closely approximates the actual data, a high-precision number of individuals can be estimated.

[0064] like Figure 11As shown, the density of the FLK group decreases during feeding. Therefore, the number of individuals can also be estimated based on images taken during feeding. In this case, the number of overlapping and invisible individuals (OBs) is reduced, making them easier to count. Even with... Figure 9 When comparing the simulated images SI shown, it is easy to compare them because the group FLK is not dense.

[0065] 5-2. Individual Model

[0066] Figure 12 This is a diagram representing an example of individual model 24.

[0067] An individual model 24 is generated based on the body structure of the individual OB. For example, a musculoskeletal model (SBM) is used as the individual model 24. The musculoskeletal SBM is obtained by dissecting an individual OB with a standard shape and size and analyzing the attachment patterns of the individual OB's bones and muscles to the bones. Figure 12 The diagram shows the musculoskeletal skeleton (SBM) of a sturgeon. A baseline model (RM) for an individual OB is reproduced based on the SBM. One or more variant models (MM1 to MMn, where n is an integer greater than or equal to 1) are derived from the baseline model RM. The baseline model RM and the variant models MM1 to MMn are collectively referred to as individual models 24. Individual models 24 are used for classification information of individual OBs.

[0068] A variation model is a three-dimensional model of an individual OB exhibiting morphological abnormalities. It is well known that almost all OBs born through captivity possess physical abnormalities (e.g., fin curvature, fin shape variations, internal organ diseases). Therefore, variation models for individual OBs can be generated for, for example, those related to pectoral fin abnormalities (e.g., pectoral fin curvature), dorsal fin abnormalities (e.g., dorsal fin curvature), caudal fin abnormalities (e.g., caudal fin curvature), and outline variations (e.g., head outline variations compared to typical individuals). Furthermore, variation models are not limited to the models mentioned above. The specific variation model generated depends on the type of OB and its environment.

[0069] Figure 13 It is a diagram representing the motion simulation of an individual OB.

[0070] The motion features obtained through motion simulation are correlated with individual model 24. In the sturgeon example, swimming method features (motion features) are extracted for each individual model 24 through fluid simulation. The swimming method of an individual OB varies according to its morphological characteristics. For example, an individual OB with physical abnormalities swims to compensate for these abnormalities. Therefore, the swimming method features of each individual model 24 are different. The swimming method features can be used as parametric information to identify individual OBs. The parametric information representing the motion features of an individual OB is correlated with individual model 24 and used as classification information.

[0071] Figure 14 It is a picture showing sturgeon raised in a fishpond.

[0072] Almost all sturgeon in fish farms exhibit some form of physical abnormality. For example, individuals OB1 through OB4 differ in the shape and slightly different outlines of their pectoral, dorsal, and caudal fins. Therefore, a variation model corresponding to the physical abnormality can be assigned to each sturgeon. Even sturgeon assigned the same variation model exhibit varying degrees of physical abnormality. Consequently, the parameters representing the motion characteristics of individual OBs also differ from sturgeon to individual. Therefore, if the parameter information of each individual OB is detected through motion simulation, each individual OB can be identified based on the detected parameter information.

[0073] 6. Structure of Information Processing Device

[0074] Figure 15 This is a diagram illustrating an example of an information processing apparatus 1 to which the information processing of this disclosure applies.

[0075] Information processing device 1 is a dedicated or general-purpose computer system such as a server. The DNN used for image analysis described above is included in analysis model 21. Information processing device 1 has a processing unit 10 and a storage unit 20. The processing unit 10 includes, for example, an input data generation unit 11, a correct data generation unit 12, an image analysis unit 13, a simulation unit 14, and a calculation unit 15.

[0076] The simulation unit 14 generates a three-dimensional simulation image SI using the simulation model specified in the model information 22. The simulation image SI uses CG technology to reproduce the simulation results as an image observed from a certain camera angle. The model information 22 specifies, for example, a group model 23, an individual model 24, and an abnormal behavior model 25. The abnormal behavior model 25 is a simulation model representing the abnormal behavior of an individual OB. The simulation unit 14 generates simulation images SI of the group FLK and the individual OB based on these simulation models.

[0077] The input data generation unit 11 generates the parameter information required for the simulation and outputs it to the simulation unit 14. The input data generation unit 11 uses the simulation unit 14 to generate a simulation image SI as the input data TID through computer graphics.

[0078] The correct data generation unit 12 generates label information based on the parameter information used in the simulation. The correct data generation unit 12 generates correct data TOD including the simulation image SI and label information. The correct data generation unit 12 can generate image elements IE for identifying individual OBs based on the location information of the individual OBs used in the simulation. In this case, the correct data generation unit 12 generates a corrected image CSI with image elements IE appended to the simulation image SI, and then appends label information to the corrected image CSI to generate the correct data TOD.

[0079] A large amount of training data is generated by the input data generation unit 11 and the correct data generation unit 12. An analysis model 21 is generated using machine learning that employs the input data TID and the correct data TOD as training data. The analysis model 21 includes, for example, a DNN for parsing the classification and location information of individual OBs, a DNN for parsing the parameter information of the group FLK, a DNN for parsing the parameter information of individual OBs, a DNN for removing reflective images MIR, and a DNN for removing shadows SH. These DNNs undergo appropriate learning corresponding to the required functions. Various machine learning methods are available, including RNN (Recurrent Neural Network), Deep Reinforcement Learning, and Transformers.

[0080] The image analysis unit 13 performs image analysis on the captured image CI using the analysis model 21. The image analysis unit 13 extracts label information from the captured image CI through image analysis. Based on the extracted label information, the image analysis unit 13 extracts the necessary parameter information.

[0081] The calculation unit 15 outputs the parameter information extracted by the image analysis unit 13 to the simulation unit 14. The calculation unit 15 uses the simulation unit 14 to generate a simulated image SI based on the extracted parameter information. The calculation unit 15 calculates the difference between the simulated image SI and the captured image CI. If the difference does not meet a preset condition, the calculation unit 15 corrects the parameter information and generates the simulated image SI. The calculation unit 15 repeatedly corrects the parameter information and generates the simulated image SI until the difference meets the above conditions. Based on the corrected parameter information that meets the above conditions, the calculation unit 15 calculates the required information.

[0082] Storage device 20 stores, for example, the program 29 executed by processing device 10, analysis model 21, and model information 22. Program 29 is a program that enables a computer to perform the information processing of this disclosure. Processing device 10 performs various processes according to program 29 stored in storage device 20. Storage device 20 can also be used as a working area for temporarily storing the processing results of processing device 10. Storage device 20 includes, for example, any non-transitory storage medium such as semiconductor storage medium and magnetic storage medium. Storage device 20 is configured, for example, as an optical disk, optical disk, or flash memory. Program 29 is stored, for example, in a non-transitory storage medium that can be read by a computer.

[0083] The processing device 10 is, for example, a computer consisting of a processor and memory. The memory of the processing device 10 includes RAM (Random Access Memory) and ROM (Read Only Memory). The processing device 10 functions as an input data generation unit 11, a correct data generation unit 12, an image analysis unit 13, an analog unit 14, and a calculation unit 15 by executing program 29.

[0084] 7. Estimation and processing of the number of individuals

[0085] Figure 16 This is a diagram illustrating an example of the calculation process for the number of individuals in the group FLK.

[0086] Analysis model 21 includes DNNA, DNNB, DNNC, and DNND. DNNA is a DNN used to remove the reflective image (MIR). DNNB is a DNN used to remove the shadow (SH). DNNC is a DNN used to extract the location and classification information of individual objects (OBs). DNND is a DNN used to extract the parameter information of the group (FLK).

[0087] The simulated image SI used for training DNND is a simulated image of the FLK group generated using swarm model 23. The label information of the DNND training data is generated based on the parameter information of the FLK group applicable to swarm model 23. If the captured image CI of the FLK group is input into DNND, the parameter information of the FLK group is extracted. DNNC extracts information such as the type, position, and size of each individual OB from the captured image CI. DNNA and DNNB separate the noise components (reflection image MIR, shadow SH) contained in the captured image CI.

[0088] Image analysis unit 13 extracts label information from the captured image CI of the FLK cluster using analysis model 21. Image analysis unit 13 extracts parameter information of the FLK cluster based on the extracted label information. Estimation unit 15 estimates the number of individual OBs contained in the FLK cluster based on the parameter information of the FLK cluster.

[0089] In the above-described estimation steps, firstly, the estimation unit 15 generates a simulated image SI of the FLK group based on information related to the parameter information of the FLK group, the type, location, and size of individual OBs. The estimation unit 15 then corrects the parameter information of the FLK group until the difference (diff) between the simulated image SI and the captured image CI satisfies a preset condition. The difference (diff) is calculated based on the difference between the feature values ​​of the captured image CI and the feature values ​​of the simulated image SI. The estimation unit 15 then estimates the number of individual OBs based on the parameter information of the FLK group that has been corrected to meet the above conditions.

[0090] The simulated image SI generated in the extrapolation step contains the reflected image MIR and shadow SH of individual OB. The number of reflected images MIR and the density of shadow SH can be clues for obtaining information about the group FLK used to obtain the blind spot location of the camera. Therefore, the difference diff also includes the difference between the reflected image MIR and shadow SH contained in the simulated image SI and the reflected image MIR and shadow SH contained in the captured image CI. By further considering the information of the difference between the reflected image MIR and shadow SH, a high-precision number of individuals can be extrapolated.

[0091] exist Figure 16 In the example, images taken during non-feeding periods are used as the captured image CI and the simulated image SI. However, the captured image CI and the simulated image SI can also be used as images of the swarm FLK during feeding. During feeding with a predator model applied, the density of the swarm FLK is lower compared to non-feeding without a predator model applied. Therefore, there are fewer overlapping and invisible individual OBs. Thus, accurately detecting the difference between the simulated image SI and the captured image CI also improves the accuracy of individual count estimation.

[0092] Furthermore, the DNNC detects the type and size of individual OBs. Therefore, the information processing device 1 may also include an input unit for inputting the type and size of the objects being counted. Thus, the user can count only individual OBs of a specific type or size.

[0093] 8. Individual tracking

[0094] Figure 17 This is a diagram illustrating an example of individual tracking.

[0095] Analysis model 21 includes DNNA, DNNB, DNNC, and DNNE. DNNE is a DNN that adds image elements (IEs) to each individual OB based on the location and classification information extracted through DNNC.

[0096] Image analysis unit 13 applies the captured image CI in analysis model 21, and adds image elements IE to each individual OB presented on the captured image CI. Figure 17 In the example, a different color element is attached as an image element IE to each individual OB. The color of an individual OB varies depending on its size and shape. The size of an individual OB is extracted from its position information. The shape of an individual OB is extracted from its classification information (morphological features). The image parsing unit 13 detects multiple individual OBs based on the image elements IE attached to them and tracks each individual OB individually.

[0097] 9. Individual identification and identification of abnormal behavior

[0098] Figure 18 This diagram illustrates an example of methods for identifying individuals and detecting abnormal behavior.

[0099] Analysis model 21 includes DNNA, DNNB, DNNC, and DNNF. DNNF is a DNN used to extract parameter information of individual OBs.

[0100] The simulated image SI used for training DNNF is a simulated image of an individual OB generated using individual model 24. Label information for the DNNF training data is generated based on the parameter information of the individual OB applied to individual model 24. If an image CI of an individual OB is input into DNNF, parameter information corresponding to the morphological features (individual model 24) of the individual OB is extracted.

[0101] Image analysis unit 13 uses analysis model 21 to extract label information for each individual OB from the captured image CI. Based on the label information, image analysis unit 13 extracts parameter information for each individual OB. The extracted parameter information includes, for example, muscle parameters applicable to the musculoskeletal model of the individual OB, and parameters representing motion feature quantities associated with the individual model 24. Inference unit 15 identifies each individual OB based on its parameter information.

[0102] In the above-described identification steps, firstly, the calculation unit 15 generates a simulated image SI of the individual OB based on its parameter information. The calculation unit 15 then corrects the parameter information of the individual OB until the difference between the simulated image of the individual OB and the captured image CI satisfies a preset condition. The difference (diff) is calculated based on the difference between the feature values ​​of the captured image CI and the feature values ​​of the simulated image SI. The calculation unit 15 identifies the individual OB based on the corrected parameter information that satisfies the above condition.

[0103] Abnormal behavior can be identified by simulating the retinal parameters of individual OBs. Retinal parameters, for example, represent the relationship between the direction of an event observed from the individual OB and the direction of the OB's movement. For instance, if the OB moves in a direction completely unrelated to the direction of the bait when it falls in front of it, an abnormality can be inferred. Furthermore, sturgeon with internal diseases sometimes show no interest in bait, exhibiting abnormal behavior such as unnaturally floating near the surface. In such cases, the captured image CI is input into the abnormal behavior model 25, and retinal parameters are extracted. By comparing these retinal parameters with pre-set benchmark values, the abnormality of the individual OB can be detected.

[0104] Detecting abnormal behavior can also help identify poor health caused by overfeeding. Therefore, it is possible to adjust or optimize feeding amounts based on the results of abnormal behavior detection.

[0105] 10. Information Processing Methods

[0106] Figure 19 This is a flowchart illustrating an example of the information processing method disclosed herein.

[0107] In step S1, the input data generation unit 11 generates a simulated image SI as input data TID using CG.

[0108] In step S2, the correct data generation unit 12 generates label information based on the parameter information used in the simulation.

[0109] In step S3, the correct data generation unit 12 generates correct data TOD, which includes the simulated image SI and label information.

[0110] In step S4, the processing device 10 generates an analysis model 21 based on machine learning using the input data TID and the correct data TOD as training data.

[0111] In step S5, the image analysis unit 13 uses the analysis model 21 to extract tag information from the captured image CI.

[0112] In step S6, the image parsing unit 13 extracts the required parameter information from the extracted label information.

[0113] In step S7, the parameter information extracted by the image analysis unit 13 is output to the simulation unit 14. The calculation unit 15 uses the simulation unit 14 to generate a simulated image SI based on the extracted parameter information.

[0114] In step S8, the calculation unit 15 calculates the difference between the simulated image SI and the captured image CI. If the difference does not meet the preset conditions, the parameter information is corrected and the simulated image SI is generated. The calculation unit 15 repeatedly corrects the parameter information and generates the simulated image SI until the difference meets the above conditions.

[0115] In step S9, the calculation unit 15 calculates the required information based on the parameter information that has been corrected to meet the above conditions.

[0116] 11. Hardware Structure

[0117] Figure 20 This is a diagram illustrating an example of the hardware structure of the information processing device 1.

[0118] Information processing device 1, for example, via Figure 20 The computer system 1000 shown is implemented. The computer system 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600 and media interface (I / F) 1700.

[0119] The CPU 1100 operates according to the programs stored in the ROM 1300 or HDD 1400, controlling various components. The ROM 1300 stores the boot program executed by the CPU 1100 when the computer system 1000 starts up, as well as programs that depend on the hardware of the computer system 1000.

[0120] HDD 1400 stores the program executed by CPU 1100 and the data used by that program. Communication interface 1500 receives data from other devices via communication network NW and sends it to CPU 1100, and sends data generated by CPU 1100 to other devices via communication network NW.

[0121] The CPU 1100 controls output devices such as monitors and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. Additionally, the CPU 1100 outputs generated data to the output devices via the input / output interface 1600.

[0122] The media interface 1700 reads the program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 into the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as DVD (Digital Versatile Disc), PD (Phase Change Rewritable Disk), an optical-magnetic recording medium such as MO (Magneto-Optical Disk), tape media, magnetic recording media, or semiconductor memory.

[0123] For example, when computer system 1000 functions as the information processing apparatus 1 of this disclosure, the CPU 1100 of computer system 1000 performs the functions of processing apparatus 10 by executing programs loaded on RAM 1200. Data in storage device 20 is stored in HDD 1400. The CPU 1100 of computer system 1000 reads and executes these programs from recording medium 1800; however, as another example, these programs may also be obtained from other devices via communication network NW.

[0124] 12. Effects

[0125] As described above, the information processing method of this disclosure includes an input data generation step and a correct data generation step. The input data generation step generates a simulated image SI as input data TID using computer graphics. The correct data generation step generates label information based on parameter information used in the simulation and generates correct data TOD including the simulated image and label information. The procedure 29 of this disclosure enables a computer to perform the above-described information processing. The information processing apparatus 1 of this disclosure performs the above-described information processing.

[0126] Based on this structure, the correct label information for the TOD data is generated using the parameter information used in the simulation. Therefore, the label information has high accuracy. Consequently, it is easy to generate high-precision training data.

[0127] The information processing method disclosed herein includes an analysis model generation step. This step generates an analysis model 21 using machine learning with the input data TID and the correct data TOD as training data.

[0128] According to this structure, the coordinate information of the CG used to generate the input data TID is reflected in the label information. Therefore, the position, shape, size of the individual OB and the parameter information of the group FLK presented on the captured image CI are accurately extracted by the analysis model 21 obtained through training.

[0129] The information processing method disclosed herein includes a tag information extraction step, a parameter information extraction step, a simulated image generation step, a parameter information correction step, and a calculation step. The tag information extraction step uses analysis model 21 to extract tag information from the captured image CI. The parameter information extraction step extracts parameter information based on the tag information. The simulated image generation step generates a simulated image SI based on the parameter information. The parameter information correction step corrects the parameter information until the difference between the simulated image and the captured image CI meets a preset condition. The calculation step calculates information based on the corrected parameter information that meets the above conditions.

[0130] Based on this structure, the parameter information extracted using analysis model 21 is corrected according to the difference between the simulated image SI and the captured image CI. Therefore, parameter information that matches the actual data (captured image CI) well is obtained. Furthermore, since the required information is inferred based on the corrected parameter information, the accuracy of information inference is improved.

[0131] The simulated image SI is a simulated image of the FLK (Fluid Group K) generated using FLK-based statistical properties. Label information is generated based on the parameter information of the FLK applied to FLK in FLK-based ...

[0132] Based on this structure, the parameter information of the FLK group can be accurately extracted from the CI image captured by the FLK group.

[0133] The information processing method disclosed herein includes a tag information extraction step, a parameter information extraction step, and an estimation step. The tag information extraction step uses analysis model 21 to extract tag information from the captured images CI of the FLK cluster. The parameter information extraction step extracts the parameter information of the FLK cluster based on the extracted tag information. The estimation step estimates the number of individual OBs contained in the FLK cluster based on the parameter information of the FLK cluster.

[0134] Based on this structure, the number of individuals in group FLK can be accurately estimated.

[0135] The estimation process includes a simulated image generation step, a parameter information correction step, and a fixed-number estimation step. The simulated image generation step generates a simulated image SI of the FLK group based on its parameter information. The parameter information correction step corrects the FLK group's parameter information until the difference between the simulated image SI and the captured image CI meets a preset condition. The fixed-number estimation step estimates the number of individual OBs based on the corrected FLK group parameter information that meets the above condition.

[0136] Based on this structure, the parameter information extracted using analysis model 21 is corrected according to the difference between the simulated image SI and the captured image CI. Therefore, parameter information that matches the actual data (captured image CI) is obtained. Furthermore, since the number of individuals is estimated based on the corrected parameter information, the number of individuals is accurately estimated.

[0137] The simulated image SI of the FLK group is a simulated image of the FLK group during feeding.

[0138] According to this structure, the density of the FLK population is lower during feeding when a predator model is applied compared to non-feeding conditions without a predator model. Therefore, there are fewer overlapping and invisible individuals (OBs). This leads to more accurate detection of the difference between the simulated image SI and the captured image CI, and also improves the accuracy of individual count estimation.

[0139] The simulated image SI is a simulated image of an individual OB generated using an individual model 24 based on the morphological features of the individual OB. Label information is generated based on the parameter information of the individual OB applicable to the individual model 24.

[0140] Based on this structure, the parameter information of an individual OB can be accurately extracted from the captured image CI of the individual OB.

[0141] The information processing method disclosed herein includes a label information extraction step, a parameter information extraction step, and an individual identification step. The label information extraction step uses analysis model 21 to extract label information for each individual OB from the captured image CI. The parameter information extraction step extracts parameter information for each individual OB based on its label information. The individual identification step identifies each individual OB based on its parameter information.

[0142] Based on this structure, individual OBs can be accurately identified.

[0143] The identification process includes a simulated image generation step, a parameter information correction step, and an individual identification step. The simulated image generation step generates a simulated image SI of the individual OB based on its parameter information. The parameter information correction step corrects the individual OB's parameter information until the difference between the simulated image SI and the captured image CI meets a preset condition. The individual identification step identifies the individual OB based on the corrected parameter information that meets the above condition.

[0144] Based on this structure, the parameter information extracted using analysis model 21 is corrected according to the difference between the simulated image SI and the captured image CI. Therefore, parameter information that matches the actual data (captured image CI) is obtained. Furthermore, since individual OBs are identified based on the corrected parameter information, the identification accuracy of individual OBs is high.

[0145] The parameter information of an individual OB includes the parameter information of the muscle parameters of the musculoskeletal model applicable to the individual OB, as well as the parameter information representing the motion feature quantities of the individual OB.

[0146] Based on this structure, individual OBs can be accurately identified by both their morphological and behavioral characteristics.

[0147] The information processing method disclosed herein includes an anomaly detection step. The anomaly detection step inputs the captured image CI of an individual OB into the abnormal behavior model 25 and detects anomalies in the individual OB.

[0148] Based on this structure, the health status of the identified individual OB can be determined.

[0149] The correct data generation process includes an image element generation step, a corrected image generation step, and a data generation step. The image element generation step generates image elements (IEs) for identifying individual OBs based on their location information used in the simulation. The corrected image generation step generates a corrected image (CSI) with the image elements IEs appended to the simulated image (SI). The data generation step appends label information to the corrected image (CSI) and generates the correct data (TOD).

[0150] Based on this structure, image elements (IEs) are precisely appended to the appropriate positions on the simulated image SI. Furthermore, the appending of image elements (IEs) is performed automatically, thus facilitating the generation of accurate data TOD.

[0151] Image elements include frame elements that surround an individual OB or color elements that use color to identify an individual OB.

[0152] Based on this structure, individual OBs are clearly separated from other image regions.

[0153] The information processing method disclosed herein includes an image element appending step and a tracking step. The image element appending step applies a captured image CI to the analysis model 21, appending image elements IE to each individual OB presented in the captured image CI. The tracking step tracks each individual OB individually based on the image elements IE appended to each individual OB.

[0154] Based on this structure, it is easy to use the image element IE as a clue for tracking.

[0155] 13. Other

[0156] The constituent elements of the devices shown in the diagrams are functional concepts and do not necessarily need to be physically configured as illustrated. That is, the specific distribution / integration of the devices is not limited to those shown in the diagrams. They can be configured by functionally or physically distributing / integrating all or part of them in any unit, depending on various loads and usage conditions.

[0157] The embodiments of this application have been described in detail above with reference to some accompanying drawings. However, these are illustrative and can be implemented in various modifications and other ways based on the manner described in the Summary of the Invention section, and according to the knowledge of those skilled in the art.

[0158] In the above embodiments, a method for generating training data using CG was described; however, this method is not limited to the examples described above and helps improve the recognition rate of all computer vision applications. The method of using CG for sophisticated musculoskeletal simulation is not limited to sturgeon but can also be applied to humans and horses. Therefore, the above-described method using AI (Artificial Intelligence) can be widely applied to a wide variety of fields.

Claims

1. An information processing method, executed by a computer, comprising the following steps: Simulated images are generated using computer graphics as input data; Generate label information based on the parameter information used in the simulation; Generate correct data including the simulated image and the label information; A machine learning analysis model is generated by using the input data and the correct data as training data. The analysis model is used to extract the label information from the captured images; Extract parameter information based on the label information; A simulated image is generated based on the parameter information; The parameter information is corrected until the difference between the simulated image and the captured image meets a preset condition; Information is deduced based on the parameter information that has been corrected to meet the conditions.

2. The information processing method according to claim 1, wherein, The simulated image is a simulated image of the population generated using a population model based on the statistical properties of the population. The label information is generated based on the parameter information of the group applicable to the group model.

3. The information processing method according to claim 2, wherein, Includes the following steps: The analysis model is used to extract the tag information from the captured images of the group; Extract the parameter information of the group based on the tag information; as well as The number of individuals in the group is estimated based on the group's parameter information.

4. The information processing method according to claim 3, wherein, The calculation steps include: A simulated image of the group is generated based on the parameter information of the group; The parameter information of the group is corrected until the difference between the simulated image of the group and the captured image meets a preset condition; and The number of individuals is estimated based on the parameter information of the group that has been corrected to meet the conditions.

5. The information processing method according to claim 3 or 4, wherein, The simulated image of the swarm is a simulated image of the swarm during feeding.

6. The information processing method according to claim 1, wherein, The simulated image is a simulated image of the individual generated using an individual model based on the individual's morphological features. The label information is generated based on the parameter information of the individual applicable to the individual model.

7. The information processing method according to claim 6, wherein, Includes the following steps: The analysis model is used to extract the label information of each individual from the captured images; Based on the individual's tag information, extract the individual's parameter information for each individual; and Each individual is identified based on its parameter information.

8. The information processing method according to claim 7, wherein, The identification steps include: A simulated image of the individual is generated based on the individual's parameter information; The parameter information of the individual is corrected until the difference between the simulated image and the captured image of the individual meets a preset condition. The individual is identified based on parameter information of the individual that has been corrected to meet the conditions.

9. The information processing method according to claim 8, wherein, The individual's parameter information includes muscle parameters applicable to the individual's musculoskeletal model and parameters representing the individual's motion characteristic quantities.

10. The information processing method according to any one of claims 7 to 9, wherein, Includes the following steps: The images of the individual are used to model abnormal behavior and detect abnormalities in the individual.

11. The information processing method according to any one of claims 6 to 9, wherein, The steps for generating the correct data include: Based on the location information of the individual used in the simulation, image elements for identifying the individual are generated. Generate a corrected image by appending the image elements to the simulated image; and The label information is attached to the corrected image, and the correct data is generated.

12. The information processing method according to claim 11, wherein, The image elements include frame elements surrounding the individual or color elements used to identify the individual using color.

13. The information processing method according to claim 11, wherein, include: The analysis model applies captured images, and image elements are appended to each individual presented in the captured images. Each individual is tracked separately based on the image elements attached to it.

14. A computer-readable, non-transitory storage medium storing a program that enables a computer to perform the following steps: Simulated images are generated using computer graphics as input data; Generate label information based on the parameter information used in the simulation; Generate correct data including the simulated image and the label information; A machine learning analysis model is generated by using the input data and the correct data as training data. The analysis model is used to extract the label information from the captured images; Extract parameter information based on the label information; A simulated image is generated based on the parameter information; The parameter information is corrected until the difference between the simulated image and the captured image meets a preset condition; Information is deduced based on the parameter information that has been corrected to meet the conditions.

15. An information processing apparatus, wherein, include: The input data generation unit generates simulated images as input data using computer graphics. as well as The correct data generation unit generates label information based on the parameter information used in the simulation, and generates correct data including the simulation image and the label information. The image analysis unit generates an analysis model based on machine learning using the input data and the correct data as training data, extracts the label information from the captured image using the analysis model, and extracts parameter information based on the label information. The calculation unit generates a simulated image based on the parameter information; The parameter information is corrected until the difference between the simulated image and the captured image meets a preset condition; Information is deduced based on the parameter information that has been corrected to meet the conditions.

Images