A target detection method, device and storage medium

Abstract

The application discloses a target detection method and device and a storage medium. The method comprises the following steps: acquiring image data to be detected; using a preset target detection model to identify the image data to be detected, and obtaining description information of the image data to be detected; the description information at least represents position information of a target detection object contained in the image data to be detected; wherein the preset target detection model is obtained by training based on a dimension resolution matrix model and a first training sample set; the dimension resolution matrix model comprises a generator, a discriminator and a detector; the first training sample set comprises at least one training sample; the training sample at least comprises target frame point array data determined based on heat map data, and the heat map data is obtained by performing first processing on a grating projection image collected by an automatic data collection device.

Description

Technical Field

[0001] This invention relates to the field of Internet technology (IT) applications, and more particularly to a detection method, apparatus, and storage medium. Background Technology

[0002] With the development of computer technology and machine learning technology, object detection technology has been applied in various industries and is one of the most widely used practical technologies. However, the construction of existing object detection data still mainly relies on manual labor, which results in high manpower and time costs for annotation. Furthermore, the different types and environments of detection data also pose challenges to the efficiency and accuracy of object detection.

[0003] For example, in the fattening pig breeding process, pen-raising is generally adopted. In large pig farms, due to the large number of pigs, manual counting is labor-intensive and prone to errors. Therefore, using artificial intelligence object detection algorithms for intelligent pig counting is one of the mainstream automation solutions. On the other hand, the prerequisites for pig re-identification (ReID), pig health status recognition, and water play recognition in the production process are all object detection. Accurate object detection is required to achieve tasks such as pig re-identification. In existing technologies, the annotation content of pig object detection datasets includes bounding box annotation and category annotation. Bounding box annotation is obtained by manually drawing the maximum bounding box of the pig outline to obtain the coordinates of the upper left and lower right corners. Category annotation is mainly set for multi-task scenarios of object detection and classification. The category setting is related to the classification task setting and generally has only one category, pig. Existing annotation methods generally include manual annotation and automated annotation-assisted strategies. Manual annotation involves manually annotating the bounding boxes and categories of all data, which is labor-intensive. The specific steps of automated annotation assistance include: First, manually annotating a portion of the data; second, training the target detection task using the partial data to obtain an intermediate model; third, using the intermediate model to predict the bounding boxes and categories of unannotated data; fourth, manually verifying the accuracy of the predictions, and if incorrect or the bounding boxes are inaccurate, performing manual correction and calibration; fifth, retraining the model using the corrected data, repeating this process until the model meets actual production needs. Existing methods for constructing target detection data for pigs still primarily rely on manual labor, with only simple automated assistance, resulting in high manpower and time costs for annotation. Furthermore, to ensure model effectiveness, existing technologies require re-collecting and re-annotating data for new pig farms (or pigpens), seasons, regions, and pig species, depending on the specific circumstances.

[0004] Therefore, there is a need for a method that has low annotation costs and can detect target objects of different types and in different environmental backgrounds. Summary of the Invention

[0005] In view of this, the main objective of the present invention is to provide a detection method, apparatus and storage medium.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] This invention provides a detection method, the method comprising:

[0008] Acquire the image data to be detected;

[0009] The image data to be detected is identified using a preset target detection model to obtain descriptive information for the image data to be detected; the descriptive information at least characterizes the location information of the target object contained in the image data to be detected.

[0010] The preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes: a generator, a discriminator, and a detector;

[0011] The first training sample set includes at least one training sample; the training sample includes at least target bounding box dot matrix data determined based on heatmap data, wherein the heatmap data is obtained by first processing a raster projection image acquired by an automated data acquisition device.

[0012] The method in the above scheme further includes:

[0013] Obtain a first training sample set; the first training sample set includes: at least one training sample, and a category label corresponding to each training sample; each training sample includes: an RGB image and target bounding box bitmap data;

[0014] The dimensionality resolution matrix model is trained based on the first training sample set to obtain the trained dimensionality resolution matrix model.

[0015] Based on the target bounding box dot matrix data corresponding to the at least one training sample and a preset blank image, generate sample target bounding box dot matrix data;

[0016] Based on the generator in the trained dimensionality resolution matrix model and the sample target bounding box dot matrix data, model inference is performed to generate at least one image data; based on at least one image data and the location data and category label corresponding to each image data, a second training sample set is determined;

[0017] The target detection model is obtained by performing feature fusion training based on the detector in the trained dimensionality resolution matrix model and the second training sample set.

[0018] In the above solution, the automated data acquisition device includes: at least one bracket, and at least one RGB camera and at least one grating projection device placed on each of the brackets;

[0019] Obtaining training samples from the first training sample set includes:

[0020] Capture RGB images using the RGB camera;

[0021] When using the grating projection device to perform grating projection, the RGB camera is used to capture images to obtain the grating projection image;

[0022] The first processing is performed on the grating projection image to obtain heat map data;

[0023] The heatmap data is processed a second time to obtain the target bounding box dot matrix data.

[0024] The method in the above scheme further includes:

[0025] The RGB image is downsampled to obtain one or more image data of different resolutions;

[0026] The step of training the dimensionality resolution matrix model based on the first training sample set to obtain the trained dimensionality resolution matrix model includes:

[0027] Based on a preset dimensionality resolution matrix model, random vectors, at least one target bounding box dot matrix data, downsampled image data corresponding to each target bounding box dot matrix data, and category labels, recurrent adversarial training is performed to obtain the trained dimensionality resolution matrix model.

[0028] In the above scheme, the dimensionality resolution matrix model includes: a first model and a second model constructed based on high and low dimensional feature directions and high and low resolution directions;

[0029] In the high and low resolution directions, both the first model and the second model include: a first part generation module, Y second part generation modules, where Y is a positive integer; and Y image data of different resolutions obtained by downsampling;

[0030] In the high and low dimension feature directions, the first model uses low-dimensional category labels, and the second model uses high-dimensional category labels;

[0031] The input data for the first part generation module includes: a random vector, image data at a first resolution, bounding box dot matrix data, and category labels;

[0032] The input data for the second part generation module includes: the image generated by the upper-level module, image data at the second resolution, target bounding box dot matrix data, and category labels; the first resolution is lower than the second resolution; the resolution of the image data used by each second part generation module is higher than the resolution of the image data used by the upper-level second part generation module;

[0033] Both the first part generation module and the second part generation module include: a generator, a discriminator, and a detector;

[0034] The step of training the dimensionality resolution matrix model based on the first training sample set includes:

[0035] The generator generates an image of a specific resolution based on a random vector and / or an image generated by a higher-level generation module; the higher-level generation module is a higher-level first part generation module or a higher-level second part generation module.

[0036] The discriminator obtains a supervision signal by distinguishing the generated image at a specific resolution from the real image, and then performs gradient backpropagation to update the generator; the real image is an RGB image or downsampled image data.

[0037] The detector reconstructs the target bounding box data from the generated image at the specific resolution, calculates the loss based on the real target bounding box data and the reconstructed target bounding box data, and performs gradient backpropagation to update the generator.

[0038] In the above scheme, generating sample target bounding box data based on the target bounding box data corresponding to the at least one training sample and a preset blank image includes:

[0039] Statistically analyze the length and width combination of the target detection object in at least one training sample;

[0040] A blank image is generated, and the width-length combination is randomly placed into the blank image. The sample target bounding box dot matrix data corresponding to the blank image in which the width-length combination is placed is determined. The blank image and the dimensionality resolution matrix model adopt the same shape.

[0041] Accordingly, the step of performing model inference based on the generator in the trained dimensionality resolution matrix model and the sample target bounding box dot matrix data to generate at least one image data point includes:

[0042] The generators in the trained dimensionality resolution matrix model are used to perform model inference on the sample target box dot matrix data to obtain the image data generated by each generator.

[0043] In the above scheme, the category labels include: high-dimensional category labels and / or low-dimensional category labels; the dimensions of the low-dimensional category labels include at least one of the following: posture, model; the dimensions of the high-dimensional category labels include at least one of the following: posture, variety, background, location of the target detection object, type of the target detection object, model.

[0044] In the above scheme, when using the grating projection device for grating projection, the RGB camera is used to capture images to obtain the grating projection image, including:

[0045] Synthesize multiple sinusoidal raster image data with different phases and periods;

[0046] The aforementioned grating projection device projects multiple sinusoidal grating image data in a time-division manner according to different projection directions and different projection cycles;

[0047] When projecting each of the sinusoidal grid image data, the RGB camera is used to capture the image after the grating is projected as the grating projection image;

[0048] Determine X grating projection images with the same projection direction, the same grating projection device, and different projection periods; where X is a multiple of 3.

[0049] The projection directions include: horizontal projection and vertical projection.

[0050] In the above scheme, the first processing based on the grating projection image to obtain heat map data includes:

[0051] RGB decomposition is performed on three grating projection images projected with the same projection direction and the same grating projection device at different periods to obtain the first matrix data; the first matrix data is 9-dimensional channel matrix data;

[0052] For each channel data of the first matrix data, perform discrete gradient calculations in two dimensions under the image coordinate system to obtain gradients in multiple directions under each coordinate, forming a gradient vector;

[0053] The gradient vector fusion algorithm is used to fuse gradient vectors of different color channels and different projection periods to obtain gradient vector data corresponding to each projection direction.

[0054] Based on the gradient vector data, determine the raster curve path with the optimal pixel granularity.

[0055] Based on the raster curve path with pixel granularity in the optimal direction, determine the target detection object location heatmap corresponding to each raster projection image;

[0056] Heatmap data is obtained by overlaying heatmaps of target locations from multiple directions.

[0057] In the above scheme, the second processing based on the heatmap data to obtain the target bounding box dot matrix data includes:

[0058] A heatmap pre-target detection model is used to detect targets in the heatmap data to obtain target detection boxes corresponding to the heatmap data; the target detection boxes are the position coordinates of the target object in the image.

[0059] The target detection box is converted into target box dot matrix data.

[0060] The above scheme further includes: generating a heatmap pre-target detection model;

[0061] The heatmap pre-target detection model includes:

[0062] Based on a preset neural network model, the at least one heatmap data and the bounding box of the target detection object corresponding to each heatmap data are trained to obtain a heatmap pre-target detection model.

[0063] This invention provides a detection device, comprising:

[0064] The acquisition module is used to acquire the image data to be detected;

[0065] The processing module is used to identify the image data to be detected using a preset target detection model and obtain descriptive information for the image data to be detected; the descriptive information at least represents the location information of the target object contained in the image data to be detected.

[0066] The preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes: a generator, a discriminator, and a detector;

[0067] The first training sample set includes at least one training sample; the training sample includes at least target bounding box dot matrix data determined based on heatmap data, wherein the heatmap data is obtained by first processing a raster projection image acquired by an automated data acquisition device.

[0068] In the above scheme, the device further includes: a training module, used to acquire a first training sample set; the first training sample set includes: at least one training sample, and a category label corresponding to each training sample; each training sample includes: an RGB image and target bounding box bitmap data;

[0069] The dimensionality resolution matrix model is trained based on the first training sample set to obtain the trained dimensionality resolution matrix model.

[0070] Based on the target bounding box dot matrix data corresponding to the at least one training sample and a preset blank image, generate sample target bounding box dot matrix data;

[0071] Based on the generator in the trained dimensionality resolution matrix model and the sample target bounding box dot matrix data, model inference is performed to generate at least one image data; based on at least one image data and the location data and category label corresponding to each image data, a second training sample set is determined;

[0072] The target detection model is obtained by performing feature fusion training based on the detector in the trained dimensionality resolution matrix model and the second training sample set.

[0073] In the above solution, the automated data acquisition device includes: at least one bracket, and at least one RGB camera and at least one grating projection device placed on each of the brackets;

[0074] The training module is used to capture RGB images using the RGB camera;

[0075] When using the grating projection device to perform grating projection, the RGB camera is used to capture images to obtain the grating projection image;

[0076] The first processing is performed on the grating projection image to obtain heat map data;

[0077] The heatmap data is processed a second time to obtain the target bounding box dot matrix data.

[0078] In the above scheme, the training module is also used to downsample the RGB image to obtain one or more image data of different resolutions;

[0079] The training module is used to perform cyclic adversarial training based on a preset dimensionality resolution matrix model, random vectors, at least one target box dot matrix data, downsampled image data corresponding to each target box dot matrix data and category labels, to obtain the trained dimensionality resolution matrix model.

[0080] In the above scheme, the dimensionality resolution matrix model includes: a first model and a second model constructed based on high and low dimensional feature directions and high and low resolution directions;

[0081] In the high and low resolution directions, both the first model and the second model include: a first part generation module, Y second part generation modules, where Y is a positive integer; and Y image data of different resolutions obtained by downsampling;

[0082] In the high and low dimension feature directions, the first model uses low-dimensional category labels, and the second model uses high-dimensional category labels;

[0083] The input data for the first part generation module includes: a random vector, image data at a first resolution, bounding box dot matrix data, and category labels;

[0084] The input data for the second part generation module includes: the image generated by the upper-level module, image data at the second resolution, target bounding box dot matrix data, and category labels; the first resolution is lower than the second resolution; the resolution of the image data used by each second part generation module is higher than the resolution of the image data used by the upper-level second part generation module;

[0085] Both the first part generation module and the second part generation module include: a generator, a discriminator, and a detector;

[0086] The generator is used to generate an image of a specific resolution based on a random vector and / or an image generated by a higher-level generation module; the higher-level generation module is a higher-level first part generation module or a higher-level second part generation module.

[0087] The discriminator is used to obtain a supervision signal by distinguishing between the generated image at a specific resolution and the real image, and to perform gradient backpropagation to update the generator; the real image is an RGB image or downsampled image data;

[0088] The detector is used to reconstruct the target bounding box data of the generated image at the specific resolution, calculate the loss based on the real target bounding box data and the reconstructed target bounding box data, and perform gradient backpropagation to update the generator.

[0089] In the above scheme, the training module is used to statistically analyze the length and width combination of the target detection object in the at least one training sample;

[0090] A blank image is generated, and the width-length combination is randomly placed into the blank image. The sample target bounding box dot matrix data corresponding to the blank image in which the width-length combination is placed is determined. The blank image and the dimensionality resolution matrix model adopt the same shape.

[0091] Accordingly, the training module is used to perform model inference on the sample target box dot matrix data using the generator in the trained dimensional resolution matrix model to obtain image data generated by each generator.

[0092] In the above scheme, the category labels include: high-dimensional category labels and / or low-dimensional category labels; the dimensions of the low-dimensional category labels include at least one of the following: posture, model; the dimensions of the high-dimensional category labels include at least one of the following: posture, variety, background, location of the target detection object, type of the target detection object, model.

[0093] In the above scheme, the training module is used to synthesize multiple sinusoidal raster image data with different phases and periods;

[0094] The aforementioned grating projection device projects multiple sinusoidal grating image data in a time-division manner according to different projection directions and different projection cycles;

[0095] When projecting each of the sinusoidal grid image data, the RGB camera is used to capture the image after the grating is projected as the grating projection image;

[0096] Determine X grating projection images with the same projection direction, the same grating projection device, and different projection periods; where X is a multiple of 3.

[0097] The projection directions include: horizontal projection and vertical projection.

[0098] In the above scheme, the training module is used to perform RGB decomposition using three grating projection images projected with different periods in the same projection direction and using the same grating projection device to obtain first matrix data; the first matrix data is 9-dimensional channel matrix data;

[0099] For each channel data of the first matrix data, perform discrete gradient calculations in two dimensions under the image coordinate system to obtain gradients in multiple directions under each coordinate, forming a gradient vector;

[0100] The gradient vector fusion algorithm is used to fuse gradient vectors of different color channels and different projection periods to obtain gradient vector data corresponding to each projection direction.

[0101] Based on the gradient vector data, determine the raster curve path with the optimal pixel granularity.

[0102] Based on the raster curve path with pixel granularity in the optimal direction, determine the target detection object location heatmap corresponding to each raster projection image;

[0103] Heatmap data is obtained by overlaying heatmaps of target locations from multiple directions.

[0104] In the above scheme, the training module is used to use the heatmap pre-target detection model to detect the heatmap data and obtain the target detection box corresponding to the heatmap data; the target detection box is the position coordinate of the target object in the image;

[0105] The target detection box is converted into target box dot matrix data.

[0106] In the above scheme, the training module is also used to generate a heatmap pre-target detection model; specifically, it is used to train the at least one heatmap data and the bounding box of the target detection object corresponding to each heatmap data according to a preset neural network model to obtain the heatmap pre-target detection model.

[0107] This invention also provides a detection device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0108] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0109] This invention provides a detection method, apparatus, and storage medium. The method includes: acquiring image data to be detected; identifying the image data using a preset target detection model to obtain descriptive information about the image data; the descriptive information at least characterizes the location information of a target object contained in the image data; wherein the preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes a generator, a discriminator, and a detector; the first training sample set includes at least one training sample; the training sample includes at least target bounding box dot matrix data determined based on heatmap data, the heatmap data being obtained by first processing a raster projection image acquired through an automated data acquisition device. Thus, the target bounding box dot matrix data generated from heatmap data achieves low-data-volume identification of the target object's location, providing data support for data construction, thereby reducing the required training samples; the target detection model obtained through the trained dimensionality resolution matrix model achieves the technical effect of robust recognition of targets in multiple dimensions. Attached Figure Description

[0110] Figure 1 A schematic flowchart of a detection method provided in an embodiment of the present invention;

[0111] Figure 2This is a schematic diagram of the layout of an automated target detection data acquisition device provided in an embodiment of the present invention;

[0112] Figure 3 for A projection schematic diagram of a time-division phase projection color grating device provided in an embodiment of the present invention;

[0113] Figure 4 This is a schematic diagram of a time-division phase projection grid device provided in an embodiment of the present invention;

[0114] Figure 5 A schematic diagram illustrating a grating projection and imaging method provided in an embodiment of the present invention;

[0115] Figure 6 A schematic diagram illustrating RGB decomposition provided in an embodiment of the present invention;

[0116] Figure 7 A schematic diagram illustrating grid direction vector calculation provided in an embodiment of the present invention;

[0117] Figure 8 A schematic diagram illustrating the calculation of optimal grid orientation provided in an embodiment of the present invention;

[0118] Figure 9 This is a schematic diagram of a heat map calculation provided in an embodiment of the present invention;

[0119] Figure 10 This is a schematic diagram of manual annotation provided in an embodiment of the present invention;

[0120] Figure 11 This is a schematic diagram illustrating the training of a pre-target detection model according to an embodiment of the present invention;

[0121] Figure 12 This is a schematic diagram illustrating a data generation method and an object detection model training method provided in an embodiment of the present invention;

[0122] Figure 13 A schematic diagram illustrating a data generation method provided in an embodiment of the present invention;

[0123] Figure 14 This is a schematic diagram illustrating the training of a target detection feature fusion layer according to an embodiment of the present invention.

[0124] Figure 15 This is a schematic diagram of the structure of a detection device provided in an embodiment of the present invention;

[0125] Figure 16 This is a schematic diagram of another detection device provided in an embodiment of the present invention. Detailed Implementation

[0126] The present invention will be further described in detail below with reference to the embodiments.

[0127] Figure 1 This is a flowchart illustrating a detection method provided in an embodiment of the present invention; as shown below. Figure 1 As shown, the method includes:

[0128] Step 101: Obtain the image data to be detected;

[0129] Step 102: Use a preset target detection model to identify the image data to be detected and obtain descriptive information for the image data to be detected; the descriptive information at least represents the location information of the target object contained in the image data to be detected.

[0130] The preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes: a generator, a discriminator, and a detector;

[0131] The first training sample set includes at least one training sample; the training sample includes at least target bounding box dot matrix data determined based on heatmap data, wherein the heatmap data is obtained by first processing a raster projection image acquired by an automated data acquisition device.

[0132] Here, the target detection model is specifically obtained by fusing features from different dimensions and detectors with different resolutions in the trained dimensional resolution matrix model.

[0133] In practical applications, the target object for detection can be any object that needs to be detected, for example, it can be used for detecting pigs.

[0134] Considering that existing object detection schemes employ automatic feature extraction via neural networks, their robustness to changes in the location, season, region, and type of the target object is poor. To enhance generalization performance, various flipping and generative algorithms are generally used for data augmentation and generation. However, existing data augmentation and generation schemes are disconnected from the object detection method, failing to achieve mutual complementarity between data generation and model training. Therefore, this application proposes to integrate the image data generation method with the object detection method based on a generative object detection image data generation method. The object detection model training (i.e., determining the object detection model) is completed during the data generation process (i.e., determining the second training sample set), improving the robustness of recognition for different scenes and different types of target objects.

[0135] Based on this, in some embodiments, the method further includes:

[0136] Obtain a first training sample set; the first training sample set includes: at least one training sample, and a category label corresponding to each training sample; each training sample includes: an RGB (RGB color mode) image and target bounding box bitmap data;

[0137] The dimensionality resolution matrix model is trained based on the first training sample set to obtain the trained dimensionality resolution matrix model.

[0138] Based on the target bounding box dot matrix data corresponding to the at least one training sample and a preset blank image, generate sample target bounding box dot matrix data;

[0139] Based on the generator in the trained dimensionality resolution matrix model and the sample target bounding box dot matrix data, model inference is performed to generate at least one image data; based on at least one image data and the location data and category label corresponding to each image data, a second training sample set is determined;

[0140] The target detection model is obtained by performing feature fusion training based on the detector in the trained dimensionality resolution matrix model and the second training sample set.

[0141] In some embodiments, the automated data acquisition device includes: at least one bracket, and at least one RGB camera and at least one grating projection device placed on each of the brackets;

[0142] Obtaining training samples from the first training sample set includes:

[0143] Capture RGB images using the RGB camera;

[0144] When using the grating projection device to perform grating projection, the RGB camera is used to capture images to obtain the grating projection image;

[0145] The first processing is performed on the grating projection image to obtain heat map data;

[0146] The heatmap data is processed a second time to obtain the target bounding box dot matrix data.

[0147] The following describes an automated data acquisition device provided by an embodiment of the present invention.

[0148] The device employs the superposition of different phases of the RGB projection grating and the use of different periods of projection gratings in different directions to achieve the positioning of the projection grating. It also generates a heat map of the target object's location by recognizing the tortuous lines of the grating.

[0149] Figure 2This is a schematic diagram of the layout of an automated target detection data acquisition device provided in an embodiment of the present invention; as shown below. Figure 2 As shown, the device includes: a grating projection device 1, an RGB camera 2, a projection device and acquisition device bracket 3, and a projection grid 4. Figure 2 In the process, each acquisition device bracket can hold an RGB camera and a grating projection device.

[0150] Figure 3 for A projection schematic diagram of a time-division phase projection color grating device provided in this embodiment of the invention; the specific operating timing is as follows: Figure 3 As shown, Figure 3 The image shows a set of data captures, which consist of 9 time-division (t=1~9) raster projection and capture operations. Each time-division operation requires the RGB camera and the raster projection device to work together under a high-precision frame synchronization module.

[0151] In some embodiments, when performing grid projection using the grating projection device, obtaining the grating projection image by capturing an image using the RGB camera includes:

[0152] Synthesize multiple sinusoidal raster image data with different phases and periods;

[0153] The aforementioned grating projection device projects multiple sinusoidal grating image data in a time-division manner according to different projection directions and different projection cycles;

[0154] When projecting each of the sinusoidal grid image data, the RGB camera is used to capture the image after the grating is projected as the grating projection image;

[0155] Determine X grating projection images with the same projection direction, the same grating projection device, and different projection periods; where X is a multiple of 3.

[0156] The projection directions include: horizontal projection and vertical projection.

[0157] Specifically, when using the grating projection device to perform grating projection, the RGB camera is used to capture images to obtain the grating projection image, including:

[0158] Step 001: Synthesize sinusoidal raster image data and project sinusoidal raster image data.

[0159] Specifically, Formula 1 is used to synthesize sinusoidal grid image data with different phases and fixed periods. After receiving the projection command from the high-precision frame synchronization module (which is connected to the RGB camera and the grating projection device respectively, so that the RGB camera and the grating projection device can shoot together), the grating projection device projects the synthesized sinusoidal grid image data.

[0160] Formula 1 is: Where pix_val is the pixel value at different grid positions (pos); P represents the phase, such as the phase of different channels; and T represents the period, such as large period, medium period, and small period.

[0161] Specifically, within the same time-division frame, different color channels are generated using different phases (P). The phase of channel R is P_R=0, and the phase of channel G is P_G= The phase of channel B, P_B = .

[0162] The period and the operating grating projection device can also be different under different time-division frames. The period T can be divided into 3 types (or more): large period T_l, medium period T_m, and small period T_s. Each grating projection device will project in all periods. For example, in time-division frame t=1, grating projection device ② is working, and T=T_s.

[0163] Step 002: Capture an RGB image;

[0164] Here, after receiving the signal from the high-precision frame synchronization module, the RGB camera performs exposure and capture to obtain a top-down image of the target detection object after the sinusoidal raster image data is projected, denoted as pic_t1. Step 002 is performed simultaneously with step 001. Considering that the exposure time of the capture and the raster projection delay may cause synchronization errors, the high-precision frame synchronization module is used to issue commands for capturing and projection to ensure that the capture and projection times are consistent.

[0165] Steps 003-009: According to the corresponding grating projection period T and the configuration of the grating projection device ( Figure 5 Projection and imaging are performed to obtain pic_t1 to pic_t9 respectively.

[0166] Here, the sinusoidal raster image data is transposed, and steps 001-009 are executed again to obtain captured images pic_t10 to pic_t18. Each step is controlled within a very short time interval, such as 30ms, and the entire capturing operation is controlled within 500ms. That is, the above steps consist of 18 steps, obtaining two sets of 18 color raster projection images, one for horizontal projection and one for vertical projection. Each set contains 9 images with different projection directions and periods, and for each image, the projected raster colors are periodically distributed, such as... Figure 5 As shown.

[0167] Figure 5 This is a schematic diagram illustrating a grating projection and imaging method according to an embodiment of the present invention; Figure 5 Only a portion of the vertical grid projection is shown; the same applies to the horizontal grid. For example... Figure 5 As shown, two cuboid protrusions (A and B) are arranged on a plane, where the height of protrusion A relative to plane C is greater than that of protrusion B. The figure illustrates three projection shots, and the schematic diagram of the three time-division grid projections is shown below. Figure 5 (1), (2), (3), among which, Figure 5 (1) The image was obtained by projecting the image from the d direction using a medium-period grid projection device ①. Figure 5 (2) The image was obtained by projecting the image from direction e using a small-period grid projection device ②. Figure 5 (3) The image was obtained by projecting from direction e using a large-period grid projection device ③. The results for other directions and periods are similar. The black areas in the image are shadows caused by radial projection. The grid at the object's position experiences a phase difference (grid curvature) due to the height and horizontal distance of the projection.

[0168] The main purposes of time-division projection here are: 1) to determine the position by encoding with different colors and time-division projection; 2) to ensure accurate edge detection of the target object in multiple directions by projection in different directions; 3) to ensure the requirements of different resolutions and scales by projecting raster data in different periods.

[0169] The above steps yield two sets of 18 color raster projection images (two sets are horizontal projection and vertical projection). The following raster decoding and location heatmap algorithms are then performed on the nine color raster projection images of the horizontal and vertical projections, respectively.

[0170] In some embodiments, the first processing based on the raster-projected image to obtain heatmap data includes:

[0171] The grating projection image is decomposed into RGB values ​​using three sinusoidal grating image data projected with the same projection direction and the same grating projection device at different periods to obtain the first matrix data; the first matrix data is a 9-dimensional channel matrix data;

[0172] Discrete gradient calculations are performed on each channel data of the first matrix data in two directional dimensions (i.e., x-direction and y-direction) in the image coordinate system to obtain gradients in multiple directions under each coordinate, forming a gradient vector;

[0173] The gradient vector fusion algorithm is used to fuse gradient vectors of different color channels and different projection periods to obtain gradient vector data corresponding to each projection direction.

[0174] Based on the gradient vector data, determine the raster curve path with the optimal pixel granularity.

[0175] Based on the raster curve path with pixel granularity in the optimal direction, determine the target detection object location heatmap corresponding to each raster projection image;

[0176] Heatmap data is obtained by overlaying heatmaps of target locations from multiple directions.

[0177] Specifically, the first matrix data is W*H*9, where W and H are the dimensions of the images captured by the RGB camera, respectively.

[0178] Specifically, the grating projection image undergoes a first processing step to obtain heat map data, including:

[0179] Step 011: Decompose the RGB data;

[0180] The grating projection data of the same direction, using the same grating projection device, but with different periods are processed. Specifically, each processing step uses three grating data images with different periods for RGB decomposition, resulting in a matrix of 9 channels with a shape of W*H*9, where W and H are the dimensions of the RGB camera images, respectively. Figure 6 A schematic diagram of its decomposition is shown.

[0181] Step 012: Calculation of grid direction vector based on discrete gradient;

[0182] For each channel of the first matrix (i.e., the matrix data of 9 channels), perform discrete gradient calculations in the x and y directions of the image coordinate system, record the gradients in multiple directions under each coordinate, and form a gradient vector.

[0183] Specifically, the calculation method uses the following formula (2):

[0184] (2)

[0185] in, Let R be the pixel value of the color channel at time t in the image coordinate system x and y. For the color R channel at time t, the gradient vector in the image coordinate system x and y is... The value of direction; This indicates calculating the gradient. To indicate direction, Represents the color R channel, time t, and direction. Calculate the gradient (differentiation, partial derivative) in the direction of the gradient. and These represent two numerical values ​​for the direction vector. The G channel and B channel are similar, so we won't go into detail here.

[0186] Calculation diagram as follows Figure 7 As shown, Figure 7 There are 8 directions in the gradient direction granularity. The direction granularity selection is a hyperparameter that can be set according to the field test. It is generally related to the camera height, ambient light interference, camera shooting quality, and projection device power.

[0187] Step 013: Using the grid direction vector fusion algorithm, data from different color channels and different projection periods are fused to obtain gradient vector data (i.e., a grid direction vector map) corresponding to each projection direction (same as the grating projection device).

[0188] The design principle of the grid direction vector fusion algorithm is as follows:

[0189] The basic principle of vector calculation in step 012 is that the grating projection device projects the object from a certain angle and then reflects it to the RGB camera. The processed data contains a lot of noise, which is why the solution of this embodiment of the invention projects from multiple angles, multiple periods, multiple phases and other means.

[0190] This section uses three time-division data points from the same grating projection device for description, consistent with the notation above. For the direction of channel R at positions x and y at time t The gradient, assuming in the direction The noise error of the upper and R channels and the gradient at time t follows a Gaussian distribution with a mean of 0. Its true distribution is denoted as... (Statistics are subject to error; these are just assumptions and are not calculated.) The distribution assumption shown in Formula 3 is derived below.

[0191] Assuming the gradient vectors of different color channels and different projected image periods (3 color channels, 3 different periods, totaling 9 channels) adopt a linear superposition model, the weights of the 9 channels are denoted as... The vector fusion model shown in Formula 4 is obtained.

[0192] Based on the above assumptions, the variance of the vector fusion model is calculated. It is assumed that the larger the variance, the more unstable the model. The model weights are obtained by minimizing the variance expression, as shown in Formula 6. In this formula, the variable subscripts and superscripts are changed to counting. Nine cycles representing different projection periods and different colors.

[0193] (3)

[0194] in, In direction When the noise error of the upper and R channels and the gradient at time t follows a Gaussian distribution with a mean of 0, its true distribution is shown; v represents noise. Indicates a Gaussian distribution;

[0195] (4)

[0196] in, This represents the fused gradient vector data of the current projection direction (projection device), i.e., the raster direction vector map. For the color R channel, time t, Position weight; Indicates time t, R channel, The pixel value at the location, t represents time, and R represents the R channel.

[0197] (5)

[0198] Here, formula (5) is a variance expression. When the variance is minimized, the weights are obtained as shown in formula (6):

[0199] (6)

[0200] in, This represents the fusion weights of the nine channels in each direction; Nine cycles representing different projection periods and different color channels; This indicates that the variance of the gradient vector in each direction is calculated for 9 channels (3 projection periods and 3 color channels).

[0201] Therefore, the grid direction vector fusion algorithm in this embodiment of the invention includes:

[0202] Using the above formula (6), calculate the variance of the gradient vector in each direction for the 9 channels (3 projection periods, 3 color channels) and determine the fusion weights in each direction for the 9 channels;

[0203] Then, the gradient vector data after fusion of the current projection direction (raster projection device) is calculated by formula (4), that is, the raster direction vector map.

[0204] Step 014: Construct a probabilistic model for the raster orientation vector map;

[0205] Here, based on the gradient vector data fused from each projection direction (projection device) in step 013, the raster curve path of pixel granularity in the optimal direction is determined. Thus, the heat map of the target detection object position corresponding to each raster projection image can be determined based on the raster curve path of pixel granularity in the optimal direction.

[0206] This invention proposes a method for solving grid curve paths, which involves solving the path by fusing gradient vector data from each projection direction (projection device) using the following steps:

[0207] Symbol convention: Establish a two-dimensional coordinate system for the image, with the origin at the lower left corner of the image, upward as the y-direction, and rightward as the x-direction; For along direction The probability value, direction Marking in the image pixel coordinate system (x,y), For the point Along direction The probability value, For direction index, The number of gradient directions. and These are the pixel width and pixel height of the captured image, respectively.

[0208] Step 41, Initialization: .in, For along direction The probability value;

[0209] Step 42: Starting from the origin of the fused gradient vector data, perform the following steps sequentially from bottom to top and left to right:

[0210] 421. Calculate the posterior probability of grid orientation:

[0211] (7)

[0212] in, For the point Along direction The probability value; l The range is from 1 to K (representing the number of gradient directions); Indicates the fused l Gradient vector data in the gradient direction, i.e., raster-oriented vector map.

[0213] 422. Calculate the prior probability of grid orientation; here it is assumed that the probability of each point appearing follows a uniform service:

[0214] (8)

[0215] (9)

[0216] in, and These are the pixel width and pixel height of the captured image, respectively;

[0217] Step 43: Determine the optimal direction for each pixel. According to Formula 10, select the direction with the highest probability as the optimal direction for point (x, y). , Figure 8 Here is an example diagram illustrating this result:

[0218] (10)

[0219] Step 045: Calculate the heatmap of the target object's location based on the raster curve path with pixel granularity in the optimal direction. This specifically includes:

[0220] Step 51: Calculate the average value of the global optimal direction according to Formula 11. , The direction is basically consistent with the direction when there is no object bending (including some noise).

[0221] (11)

[0222] in, This represents the average value of the globally optimal direction. This indicates the optimal direction for point (e, f).

[0223] Step 52: Calculate the position intensity value of the grid stripes projected vertically (the method is the same for horizontal projection) along the positive y-direction (or along the positive x-direction) by column. The position intensity value of each point corresponds to the current column sequence (row sequence). The sum of the magnitudes of the directional offset;

[0224] The calculation steps are as follows:

[0225] Step 521: Select all points without an optimal direction; such as... Figure 9 (1) The points marked in yellow are marked with an intensity value of 0;

[0226] Step 522: Starting from each point without an optimal direction, traverse all points along the optimal direction path. This will yield multiple acyclic path graphs starting from points without optimal directions (acyclicity is guaranteed by the gradient calculation direction); for example... Figure 9 (2) is shown;

[0227] Step 523: Compare each path graph with the path graph under the global average direction; for example... Figure 9 As shown in (3), the intensity of the current point is obtained by subtracting the corresponding points of the two path diagrams and taking the modulus.

[0228] Through the above steps 011-015, each grating projection image will obtain a heat map. Each heat map is obtained by projection from different projection directions (projection device). Different shadow directions will have different occlusions. The final heat map can be obtained by superimposing heat maps from multiple directions.

[0229] In some embodiments, a second processing is performed on the heatmap data to obtain target bounding box dot matrix data, including:

[0230] A heatmap pre-target detection model is used to detect targets in the heatmap data to obtain target detection boxes corresponding to the heatmap data; the target detection boxes are the position coordinates of the target object in the image.

[0231] The target detection box is converted into target box dot matrix data.

[0232] In some embodiments, the method further includes: generating a heatmap pre-target detection model;

[0233] The heatmap pre-target detection model includes:

[0234] Based on a preset neural network model, the at least one heatmap data and the bounding box of the target detection object corresponding to each heatmap data are trained to obtain a heatmap pre-target detection model.

[0235] Here, the bounding boxes of the target objects can be manually labeled in advance using the obtained heatmap data of the target object location and the RGB image data. Since the heatmap is obtained by processing RGB images, the labeling data of the RGB images can be directly applied to the heatmap. A schematic diagram of the bounding box labeling of the target objects is shown below. Figure 10 As shown in the figure, taking pigs as the target detection object, the gray squares (left image) represent other items that are not pigs (such as water troughs), which are also shown in the heat map (right image).

[0236] Based on sample heatmap data and manually labeled data (boundary boxes of target detection objects), existing technologies (such as deep learning target detection methods SSD, YOLO, etc.) are used for training and recognition to obtain a heatmap pre-target detection model.

[0237] Here, because the heatmap data is of a single format and the location of the target object has obvious features, the learning difficulty is lower than that of traditional image target detection tasks. It does not require too much data, and can achieve good results with less than 100 images in real-world scenarios. However, traditional image target detection tasks require more than 2,000 images in real-world scenarios to achieve the same effect. Figure 11 This is a schematic diagram illustrating the training of a detection model according to an embodiment of the present invention.

[0238] In some embodiments, the category labels include: high-dimensional category labels and / or low-dimensional category labels; the dimensions of the low-dimensional category labels include at least one of the following: posture, model; the dimensions of the high-dimensional category labels include at least one of the following: posture, variety, background, location of the target detection object, type of the site where the target detection object is located, model.

[0239] For example, the target detection object is pigs; the low-dimensional category label may include at least one of the following: upright large pig, lying large pig, upright small pig, lying small pig. The high-dimensional category label may include at least one of the following: upright large spotted pig, lying large spotted pig, upright small spotted pig, lying small spotted pig, upright large white pig, lying large white pig, upright small white pig, lying small white pig, upright large black pig, lying large black pig, upright small black pig, lying small black pig. The high-dimensional category label may also include at least one of the following dimensional information: background, pig farm location, pig farm type.

[0240] By using different training samples with different types of labels, and the construction method of the dimensional resolution matrix model, a method for generating image data with multi-level dimensional feature input reconstruction and multi-level resolution is adopted to achieve the technical effect of generating target detection data under various conditions such as background category, target detection object location category, target detection object type, target detection object pose, and different resolution.

[0241] The following further explains the specific process of completing the target detection model training (i.e., determining the target detection model) during the data generation process (i.e., determining the second training sample set).

[0242] In some embodiments, the method further includes:

[0243] The RGB image is downsampled to obtain one or more image data of different resolutions;

[0244] The step of training the dimensionality resolution matrix model based on the first training sample set to obtain the trained dimensionality resolution matrix model includes:

[0245] Based on a preset dimensionality resolution matrix model, random vectors, at least one target bounding box dot matrix data, downsampled image data corresponding to each target bounding box dot matrix data, and category labels, recurrent adversarial training is performed to obtain the trained dimensionality resolution matrix model.

[0246] Here, the dimensionality resolution matrix model includes: a first model and a second model constructed based on high and low dimensional feature directions and high and low resolution directions;

[0247] In the high and low resolution directions, both the first model and the second model include: a first part generation module and Y second part generation modules, where Y is a positive integer; the image data of different resolutions obtained by downsampling are Y; that is, the number of downsampling can be determined based on the number of second part generation modules.

[0248] In the high and low dimension feature directions, the first model uses low-dimensional category labels, and the second model uses high-dimensional category labels;

[0249] The input data for the first part generation module includes: a random vector, image data at a first resolution, bounding box dot matrix data, and category labels;

[0250] The input data for the second part generation module includes: the image generated by the upper-level module, image data at the second resolution, target bounding box dot matrix data, and category labels; the first resolution is lower than the second resolution; and the resolution of the image data used by each second part generation module is higher than the resolution of the image data used by the upper-level second part generation module.

[0251] In some embodiments, each of the first part generation module and the second part generation module includes: a generator, a discriminator, and a detector.

[0252] The step of training the dimensionality resolution matrix model based on the first training sample set includes:

[0253] The generator generates an image of a specific resolution based on a random vector and / or an image generated by a higher-level generation module.

[0254] The discriminator obtains a supervision signal by distinguishing the generated image at a specific resolution from the real image, and then performs gradient backpropagation to update the generator; the real image is an RGB image or downsampled image data.

[0255] The detector reconstructs the target bounding box data from the generated image at the specific resolution, calculates the loss based on the real target bounding box data and the reconstructed target bounding box data, and performs gradient backpropagation to update the generator.

[0256] Here, the superior generation module is either the superior first part generation module or the superior second part generation module; the actual target bounding box dot matrix data is the target bounding box dot matrix data input to each generation module.

[0257] The training direction is as follows: the generator generates an image at the current resolution under the conditions of the input target box bitmap data and the constraints of the discriminator, and the detector can use the generated image to restore the target box bitmap data.

[0258] In some embodiments, generating sample target bounding box data based on the target bounding box data corresponding to the at least one training sample and a preset blank image includes:

[0259] Statistically analyze the length and width combination of the target detection object in at least one training sample;

[0260] A blank image is generated, and the width-length combination is randomly placed into the blank image. The sample target bounding box dot matrix data corresponding to the blank image in which the width-length combination is placed is determined. The blank image and the dimensionality resolution matrix model adopt the same shape.

[0261] Accordingly, the step of performing model inference based on the generator in the trained dimensionality resolution matrix model and the sample target bounding box dot matrix data to generate at least one image data point includes:

[0262] The generators in the trained dimensionality resolution matrix model are used to perform model inference on the sample target box dot matrix data to obtain the image data generated by each generator.

[0263] The second training sample set is determined based on the generated image data, as well as the category label and location data corresponding to each generated image data.

[0264] Here, based on the target bounding box data corresponding to the image data in the existing training samples, all width-to-length combinations in the image data are statistically analyzed to obtain them. From all these combinations, a batch (also of random quantity) of width-to-length combinations is randomly selected, and each randomly selected combination is randomly placed into a blank image. Following the method described above for generating target bounding box bitmap data, the sample target bounding box bitmap data for the blank image containing the selected width-to-length combination is determined. Model inference is a specialized term in neural network knowledge, generally referring to the calculation and obtaining of results using a combination of neural networks with a given model structure and trained weights. It will not be elaborated upon here.

[0265] After training to obtain the dimensionality resolution matrix model and the second training sample set, feature fusion training is performed using the detector trained with the dimensionality resolution matrix model and the obtained second training sample set to obtain the target detection model.

[0266] In this design, due to the addition of dimensional feature axes, the features extracted from higher-dimensional modules are more dimensional. Therefore, high- and low-dimensional feature fusion is performed. The image data generated by the generator after model inference is used to train the feature fusion layer. Feature fusion can utilize common techniques such as fully connected layers or attention mechanisms, which will not be elaborated upon here.

[0267] To further illustrate the method of training an object detection model and generating image data, this invention provides a method for data generation and object detection model training, such as... Figure 12 and 13 As shown, the method includes:

[0268] Step 1: Preprocess the RGB images and heatmap data captured by the RGB camera.

[0269] Specifically, it includes:

[0270] Step 11, Heatmap data processing; including:

[0271] Step 111: Input the heatmap data into the heatmap pre-target detection model for detection to obtain target detection boxes. The target detection boxes are the target detection objects. Figure 12 and Figure 13 (Taking pigs as the target detection object as an example) The position coordinates in the image are generally expressed using the x and y coordinates of the top left corner of the bounding box, the width w of the bounding box, and the y coordinate of the bounding box.

[0272] Step 112: Convert the target detection bounding box into target bounding box bitmap data; including:

[0273] Step 1121: Generate blank data of the same size as the heatmap data. For example, if the heatmap data has a pixel size of 480*640, then generate a 480*640 matrix data (blank image) with all elements set to 0.

[0274] Step 1122: Set the pixel values ​​of the area within the target bounding box to 1, and the others to 0.

[0275] Step 12: RGB image data processing;

[0276] Here, step 12 includes two downsampling operations; where the original image is labeled A, the first downsampling is A', the second downsampling is A'', and so on, to obtain the image data for each sample.

[0277] The basic principle of the downsampling operation is as follows: For an image of size M*N, it is downsampled by a factor of s to obtain a resolution image of size (M / s)*(N / s). If the image is in matrix form, the image within the s*s window of the original image is transformed into a single pixel, and the value of this pixel represents all pixels within the window.

[0278] Step 13: Determine the category labels;

[0279] The category labels are divided into low-dimensional category labels and high-dimensional category labels. The label data is manually labeled during image sampling, and the data uses one-hot encoding.

[0280] For example, low-dimensional category labels include: upright large pig, lying large pig, upright small pig, and lying small pig; high-dimensional category labels include: upright large spotted pig, lying large spotted pig, upright small spotted pig, and lying small spotted pig; upright large white pig, lying large white pig, upright small white pig, and lying small white pig; upright large black pig, lying large black pig, upright small black pig, and lying small black pig. Following these examples, low-dimensional category labels distinguish between shapes, while high-dimensional category labels distinguish between shapes and colors. This method allows the neural network to acquire features across different dimensions, making generation and object detection more robust. In addition to the examples above, labels can also incorporate background, pig farm location, pig farm type, and other classification information.

[0281] Step 2: Train the dimensional resolution matrix model;

[0282] The dimensional resolution matrix model includes a first part generation module and a second part generation module. This invention relates to a method for generating images with specific resolution and dimensional features based on random vectors (the second part generation module uses a lower-resolution generated image, i.e., downsampled data) and bounding box pixel data. Its basic principle is based on bounding box feature reconstruction and recurrent adversarial training.

[0283] Each first part generation module and second part generation module includes:

[0284] Generator G is used to generate an image of a specific resolution from a lower resolution generated by a random vector or a higher resolution generator module;

[0285] Discriminator D is used to obtain a supervision signal by distinguishing the generated image at a specific resolution from the real image; the generator G is updated by gradient backpropagation (gradient backpropagation is a general technique).

[0286] The detector C is used to reconstruct the target bounding box data of the generated image and calculate the loss based on the real target bounding box data to update the generator G through gradient backpropagation.

[0287] The generator, discriminator, and classifier are constructed by stacking convolutional neural networks, pooling neural networks, and fully connected convolutional networks, respectively. The stacking method is not protected by this proposal and will not be elaborated further.

[0288] like Figure 12 As shown in step two, the model design includes two coordinate axes: the high- and low-dimensional feature directions and the high- and low-resolution directions. The models in these two directions are generated by the first part of the module (see...). Figure 12The part within the dashed box marked with α) and the second part generation module (see Figure 12 The image consists of the portion within the dashed box marked with β and the subsequent generation modules. The high and low resolution directions consist of a first part generation module and Y second part generation modules (in the example in the figure, Y=2, the number of downsampling times is 2, resulting in the first downsampled image data A' and the second downsampled image data A'').

[0289] Description of the resolution axis direction: The first part generation module and the second part generation module have similar structures, only the input is different.

[0290] The input data for the first part generation module consists of a random vector, target bounding box dot matrix data, image data A'', and category label data.

[0291] The input data for the second part generation module includes the lower-resolution image generated by the previous generation, the target bounding box dot matrix data, the image data (the first second part generation module uses image data A', and the second second part generation module uses the original RGB image data A), and the category label data.

[0292] From the perspective of the high-resolution axis, the resolution of the generated image gradually increases as it extends in the positive direction (right). Each generation module consists of a generator (G_*_, * represents the coordinates of the dimensional resolution matrix model), a discriminator (D_*_*), and a detector (C_*_*).

[0293] The training method is as follows: the generator generates an image at the current resolution under the conditions of the input target box bitmap data and the constraints of the discriminator, and the detector can use the generated image to restore the target box bitmap data.

[0294] Description of feature axis direction: Different dimensions of generated groups (vertical description) use different dimension category labels, such as Figure 12 As shown, the first row uses low-dimensional category labels, and the second group uses high-dimensional category labels. Here, the dimension of the category labels refers to the granularity of the category division, as described in step one of the labeling steps.

[0295] The specific training steps include:

[0296] Step 21: Generate random vectors. Random vectors are obtained by sampling using a Gaussian distribution, with one sampling per iteration.

[0297] Step 22: Input the random vector z, the target bounding box matrix data, and the high and low dimension category labels into the generation modules at different positions in the matrix;

[0298] Step 23: Perform model inference by level, and execute each generation module in the matrix model respectively. The generators, discriminators and detectors at each level obtain the results through neural network calculation.

[0299] Step 24: Calculate the generation loss. Inference is performed by each generation module in the matrix to obtain the generated images at each level. The generated images (referred to as G_p in the figure) and RGB image data (referred to as A, A' or A'' in the figure) are sent to the discriminator of the current group to obtain the discrimination results of the two images respectively. The label of the generated image is set to 0 and the label of the RGB image data is set to 1. The classification loss is calculated (using techniques such as cross-entropy).

[0300] Step 25: Calculate the detection loss. Generate an image (referred to as G_p in the figure) and send it to the detector. Obtain the predicted target box dot matrix data (referred to as B_p in the figure) and perform differential calculation on the target box dot matrix data (referred to as B in the figure) (mean squared error function MSE, etc. can be used). Use this as the detection loss. After the calculation is completed, add it to the step 26 and perform gradient backpropagation to update the weights of the generator and detector. This part is a public technology and will not be described in detail here.

[0301] Step 26: Calculate the discrimination loss. Again, infer the generated images at each level through the generation modules in the matrix. The generated images (referred to as G_p in the figure) and RGB image data (referred to as A, A', or A'' in the figure) are sent to the discriminator of the current group to obtain the discrimination results of the two images. Set the label of the generated images to 1 and the label of the RGB image data to 0. Calculate the classification loss (which can be calculated using techniques such as cross-entropy) and perform gradient backpropagation to update the weights of the discriminator.

[0302] Step 27: Repeat steps 21-24 until the calculated loss remains unchanged, and obtain the generation module for each matrix coordinate.

[0303] like Figure 12 For example, this would result in 2 x 3 generating modules (each dimension includes: a first part generating module). Figure 12 The first generation module of each row, the part marked with α in the dotted box) and the two second part generation modules ( Figure 12 The second generation module in each row, the part in the β-labeled dashed box, and the subsequent generation modules), where each generation module consists of 1 generator, 1 detector and 1 discriminator.

[0304] Step 3: Obtain the generated image data from each generation module, and package it with category labels and location data as the final dataset.

[0305] Specifically, step three includes:

[0306] Step 31: Target aspect ratio statistics: Based on the target bounding box data corresponding to the existing image data, perform statistics to obtain all length and width combinations in the data;

[0307] For example, the length and width of the frame for pigs can be calculated.

[0308] Step 32, Data Generation, including:

[0309] Step 321: Generate a blank image with the same shape as the one in Step 2;

[0310] Step 322: Randomly select a batch (the number is also random) of length and width combinations from the length and width combinations in step 31, and randomly place each length and width into a blank image to generate target bounding box dot matrix data in the manner of step one.

[0311] Step 33: Model Reasoning;

[0312] Specifically, Figure 13 This is a flowchart illustrating an image data generation method provided in an embodiment of the present invention, as shown below. Figure 13 As shown, the generator (G_*_* in the figure) in the generation module obtained in step two is used to perform model inference on the generated target box dot matrix data (B in the figure). (Model inference is a specialized term in neural network knowledge, which generally refers to the calculation of results by combining neural networks with a given model structure and trained weights.) The image data generated by each generation module (R in the figure) is obtained. The generated image data, category labels and location data are packaged together as the final dataset (equivalent to the second training sample set).

[0313] The data generation part is now complete.

[0314] Step 4: Use the detector (C_*_*) trained in Step 2 and the labeled image data generated in Step 3 (i.e., the second training sample set) to perform feature fusion training.

[0315] As described in step two, due to the addition of dimensional feature axes, the features extracted from higher-dimensional feature modules are more dimensional. Therefore, high- and low-dimensional feature fusion is performed, and the data from step three is used to train the feature fusion layer. Figure 14 As shown, Figure 14 This is a schematic diagram illustrating the training of a feature fusion layer for object detection according to an embodiment of the present invention. The feature fusion can utilize common techniques such as fully connected layers or attention mechanisms, which will not be elaborated upon here.

[0316] Figure 15 This is a schematic diagram of the structure of a detection device provided in an embodiment of the present invention; as shown below. Figure 15As shown, the device includes:

[0317] The acquisition module is used to acquire the image data to be detected;

[0318] The processing module is used to identify the image data to be detected using a preset target detection model and obtain descriptive information for the image data to be detected; the descriptive information at least represents the location information of the target object contained in the image data to be detected.

[0319] The preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes: a generator, a discriminator, and a detector;

[0320] The first training sample set includes at least one training sample; the training sample includes at least target bounding box dot matrix data determined based on heatmap data, wherein the heatmap data is obtained by first processing a raster projection image acquired by an automated data acquisition device.

[0321] It should be noted that the detection device provided in the above embodiments is only illustrated by the division of the above program modules when implementing the corresponding detection method. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the server can be divided into different program modules to complete all or part of the processing described above. In addition, the device and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0322] Figure 16 This is a schematic diagram of the structure of a detection device provided in an embodiment of the present invention, as shown below. Figure 16 As shown, the detection device 160 includes: a processor 1601 and a memory 1602 for storing a computer program capable of running on the processor; when the processor 1601 runs the computer program, it performs the following: acquiring image data to be detected; identifying the image data to be detected using a preset target detection model to obtain descriptive information for the image data to be detected; the descriptive information at least characterizes the location information of the target object contained in the image data to be detected; wherein, the preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes: a generator, a discriminator, and a detector; the first training sample set includes: at least one training sample; the training sample at least includes target bounding box dot matrix data determined based on heatmap data, the heatmap data being obtained after first processing of a raster projection image acquired by an automated data acquisition device. Specifically, the detection device can also perform the following: Figure 1 The method shown is the same as Figure 1 The detection method embodiments shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0323] In practical applications, the detection device 160 may further include at least one network interface 1603. The various components of the detection device 160 are coupled together via a bus system 1604. It is understood that the bus system 1604 is used to achieve communication between these components. In addition to a data bus, the bus system 1604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 16 All buses are labeled as bus system 1604. The number of processors 1601 can be at least one. Network interface 1603 is used for wired or wireless communication between the detection device 160 and other devices.

[0324] The memory 1602 in this embodiment of the invention is used to store various types of data to support the operation of the detection device 160.

[0325] The methods disclosed in the above embodiments of the present invention can be applied to processor 1601, or implemented by processor 1601. Processor 1601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 1601 or by instructions in the form of software. The processor 1601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 1602. Processor 1601 reads the information in memory 1602 and combines its hardware to complete the steps of the aforementioned method.

[0326] In an exemplary embodiment, the detection device 160 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0327] This invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it executes the following: acquiring image data to be detected; identifying the image data to be detected using a preset target detection model to obtain descriptive information about the image data to be detected; the descriptive information at least characterizes the location information of the target object contained in the image data to be detected; wherein the preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes: a generator, a discriminator, and a detector; the first training sample set includes: at least one training sample; the training sample at least includes target bounding box dot matrix data determined based on heatmap data, the heatmap data being obtained by first processing a raster projection image acquired by an automated data acquisition device. Specifically, the computer program can also execute the following: Figure 1 The method shown is the same as Figure 1 The detection method embodiments shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0328] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0329] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0330] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0331] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0332] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0333] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0334] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0335] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A target detection method, characterized in that, The method includes: Acquire the image data to be detected; The image data to be detected is identified using a preset target detection model to obtain descriptive information for the image data to be detected; the descriptive information at least characterizes the location information of the target object contained in the image data to be detected. The preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes a first model and a second model constructed based on high and low dimensional feature directions and high and low resolution directions. In the high and low resolution directions, both the first model and the second model include: a first part generation module, Y second part generation modules, where Y is a positive integer; and Y image data of different resolutions obtained by downsampling; In the high- and low-dimensional feature directions, the first model uses low-dimensional category labels, and the second model uses high-dimensional category labels; The input data for the first part generation module includes: a random vector, image data at a first resolution, bounding box dot matrix data, and category labels; The input data for the second part generation module includes: the image generated by the upper-level module, image data at the second resolution, target bounding box dot matrix data, and category labels; the first resolution is lower than the second resolution; the resolution of the image data used by each second part generation module is higher than the resolution of the image data used by the upper-level second part generation module; Both the first part generation module and the second part generation module include: The generator is used to generate an image of a specific resolution based on a random vector or an image generated by a higher-level generation module; the higher-level generation module is either the first part generation module or the second part generation module of the higher level. A discriminator is used to obtain a supervision signal by distinguishing the generated image at a specific resolution from the real image, and to perform gradient backpropagation to update the generator; the real image is an RGB image or downsampled image data; The detector is used to reconstruct the target bounding box data of the generated image at the specific resolution, calculate the loss based on the real target bounding box data and the reconstructed target bounding box data, and perform gradient backpropagation to update the generator; the real target bounding box data is the target bounding box data input to each generation module; the image at the specific resolution is the image at the current level resolution generated by the generator under the conditions of the input target bounding box data and the constraints of the discriminator. The first training sample set includes at least one training sample; the training sample includes at least target bounding box dot matrix data determined based on heatmap data, wherein the heatmap data is obtained by first processing a raster projection image acquired by an automated data acquisition device.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a first training sample set; the first training sample set includes: at least one training sample, and a category label corresponding to each training sample; each training sample includes: an RGB image and target bounding box bitmap data; The dimensionality resolution matrix model is trained based on the first training sample set to obtain the trained dimensionality resolution matrix model. Based on the target bounding box dot matrix data corresponding to the at least one training sample and a preset blank image, sample target bounding box dot matrix data is generated; the target bounding box dot matrix data corresponding to the at least one training sample is the target bounding box data corresponding to the image data in the existing training samples; the sample target bounding box dot matrix data is the sample target bounding box dot matrix data placed in a blank image with a length-width combination, and the length-width combination is a batch of length-width combinations randomly selected from all length-width combinations included in the image data in the existing training samples; Based on the generator in the trained dimensionality resolution matrix model and the sample target bounding box dot matrix data, model inference is performed to generate at least one image data; based on at least one image data and the location data and category label corresponding to each image data, a second training sample set is determined; The target detection model is obtained by performing feature fusion training based on the detector in the trained dimensionality resolution matrix model and the second training sample set.

3. The method according to claim 2, characterized in that, The automated data acquisition device includes: at least one bracket, and at least one RGB camera and at least one grating projection device placed on each of the brackets; Obtaining training samples from the first training sample set includes: Capture RGB images using the RGB camera; When using the grating projection device to perform grating projection, the RGB camera is used to capture images to obtain the grating projection image; The first processing is performed on the grating projection image to obtain heat map data; The heatmap data is processed a second time to obtain the target bounding box dot matrix data.

4. The method according to claim 2, characterized in that, The method further includes: The RGB image is downsampled to obtain one or more image data of different resolutions; The step of training the dimensionality resolution matrix model based on the first training sample set to obtain the trained dimensionality resolution matrix model includes: Based on a preset dimensionality resolution matrix model, random vectors, at least one target bounding box dot matrix data, downsampled image data corresponding to each target bounding box dot matrix data, and category labels, recurrent adversarial training is performed to obtain the trained dimensionality resolution matrix model.

5. The method according to claim 2, characterized in that, The step of generating sample target bounding box data based on the target bounding box data corresponding to the at least one training sample and a preset blank image includes: Statistically analyze the length and width combination of the target detection object in at least one training sample; A blank image is generated, and the width-length combination is randomly placed into the blank image. The sample target bounding box dot matrix data corresponding to the blank image in which the width-length combination is placed is determined. The blank image and the dimensionality resolution matrix model adopt the same shape. Accordingly, the step of performing model inference based on the generator in the trained dimensionality resolution matrix model and the sample target bounding box dot matrix data to generate at least one image data point includes: The generators in the trained dimensionality resolution matrix model are used to perform model inference on the sample target box dot matrix data to obtain the image data generated by each generator.

6. The method according to claim 2, characterized in that, The category labels include: high-dimensional category labels and / or low-dimensional category labels; the dimensions of the low-dimensional category labels include at least one of the following: posture, model; the dimensions of the high-dimensional category labels include at least one of the following: posture, variety, background, location of the target object, type of the site where the target object is located, model.

7. The method according to claim 3, characterized in that, When using the grating projection device to perform grating projection, the RGB camera is used to capture images to obtain the grating projection image, including: Synthesize multiple sinusoidal raster image data with different phases and periods; The aforementioned grating projection device projects multiple sinusoidal grating image data in a time-division manner according to different projection directions and different projection cycles; When projecting each of the sinusoidal grid image data, the RGB camera is used to capture the image after the grating is projected as the grating projection image; Determine X grating projection images with the same projection direction, the same grating projection device, and different projection periods; where X is a multiple of 3. The projection directions include: horizontal projection and vertical projection.

8. The method according to claim 3, characterized in that, The first processing based on the grating projection image to obtain heat map data includes: RGB decomposition is performed on three grating projection images projected with the same projection direction and the same grating projection device at different periods to obtain the first matrix data; the first matrix data is 9-dimensional channel matrix data; For each channel data of the first matrix data, perform discrete gradient calculations in two dimensions under the image coordinate system to obtain gradients in multiple directions under each coordinate, forming a gradient vector; The gradient vector fusion algorithm is used to fuse gradient vectors of different color channels and different projection periods to obtain gradient vector data corresponding to each projection direction. Based on the gradient vector data, determine the raster curve path with the optimal pixel granularity. Based on the raster curve path with pixel granularity in the optimal direction, determine the target detection object location heatmap corresponding to each raster projection image; Heatmap data is obtained by overlaying heatmaps of target locations from multiple directions.

9. The method according to claim 3, characterized in that, A second processing step is performed on the heatmap data to obtain target bounding box dot matrix data, including: A heatmap pre-target detection model is used to detect targets in the heatmap data to obtain target detection boxes corresponding to the heatmap data; the target detection boxes are the position coordinates of the target object in the image. The target detection box is converted into target box dot matrix data.

10. The method according to claim 9, characterized in that, The method further includes: generating a heatmap pre-target detection model; The heatmap pre-target detection model includes: Based on a preset neural network model, the at least one heatmap data and the bounding box of the target detection object corresponding to each heatmap data are trained to obtain a heatmap pre-target detection model.

11. A target detection device, characterized in that, include: The acquisition module is used to acquire the image data to be detected; The processing module is used to identify the image data to be detected using a preset target detection model and obtain descriptive information for the image data to be detected; the descriptive information at least represents the location information of the target object contained in the image data to be detected. The preset target detection model is trained based on a dimensionality resolution matrix model and a first training sample set; the dimensionality resolution matrix model includes a first model and a second model constructed based on high and low dimensional feature directions and high and low resolution directions. In the high and low resolution directions, both the first model and the second model include: a first part generation module, Y second part generation modules, where Y is a positive integer; and Y image data of different resolutions obtained by downsampling; In the high- and low-dimensional feature directions, the first model uses low-dimensional category labels, and the second model uses high-dimensional category labels; The input data for the first part generation module includes: a random vector, image data at a first resolution, bounding box dot matrix data, and category labels; The input data for the second part generation module includes: the image generated by the upper-level module, image data at the second resolution, target bounding box dot matrix data, and category labels; the first resolution is lower than the second resolution; the resolution of the image data used by each second part generation module is higher than the resolution of the image data used by the upper-level second part generation module; Both the first part generation module and the second part generation module include: The generator is used to generate an image of a specific resolution based on a random vector or an image generated by a higher-level generation module; the higher-level generation module is either the first part generation module or the second part generation module of the higher level. A discriminator is used to obtain a supervision signal by distinguishing the generated image at a specific resolution from the real image, and to perform gradient backpropagation to update the generator; the real image is an RGB image or downsampled image data; The detector is used to reconstruct the target bounding box data of the generated image at the specific resolution, calculate the loss based on the real target bounding box data and the reconstructed target bounding box data, and perform gradient backpropagation to update the generator; the real target bounding box data is the target bounding box data input to each generation module; the image at the specific resolution is the image at the current level resolution generated by the generator under the conditions of the input target bounding box data and the constraints of the discriminator. The first training sample set includes at least one training sample; the training sample includes at least target bounding box dot matrix data determined based on heatmap data, wherein the heatmap data is obtained by first processing a raster projection image acquired by an automated data acquisition device.

12. A target detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images