Image-based target detection method and device

Abstract

The invention discloses a target detection method based on an image, and the method comprises the steps: obtaining image data, carrying out the motion detection of the obtained image data, converting the three-channel data of a current image frame into two-channel data under the condition that a moving target exists, and carrying out the detection of the two-channel data, motion detection information data obtained through motion detection and the converted two-channel data are input into a trained target detection model for target detection, and the target detection model is obtained through training on the basis of motion detection information sample data and two-channel sample data with the same conversion mode as the converted two-channel data. According to the invention, under the condition that the structure of the target detection model is kept unchanged, the background area which does not generate motion in the image frame is filtered out in the target detection process, so that the difficulty of target detection is reduced, the false detection rate and the omission ratio are reduced, and the accuracy of target detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of image monitoring, and in particular to an image-based target detection method. Background Art

[0002] With the popularization of intelligent surveillance cameras, target detection has become an indispensable function, which can effectively reduce manual participation and thus realize the early warning function.

[0003] In current intelligent surveillance cameras, motion detection technology is usually used to detect moving objects in the scene. Due to the particularity of the technology used in this method (frame difference method and background modeling method), the false alarm rate of this method is very high. For example, large-scale leaf swaying and the strobe generated by fluorescent lights can cause motion detection to generate an alarm.

[0004] To solve the above problems, many smart cameras currently circumvent them through two steps:

[0005] The first step is to use motion detection technology to detect moving objects.

[0006] In the second step, after motion detection finds moving objects in the scene, the target detection method is used to detect whether there are targets of interest to the user in the image, so as to perform a second filtering, and then issue an alarm if there are targets of interest.

[0007] However, the target detection method used in the second step usually only uses the current frame image data, so the missed detection rate and false detection rate of target detection are often high, resulting in false alarms and missed alarms.

[0008] Some plug-in smart cameras use the previous and next two frames of images to input into the target detection model for detection, thereby improving the accuracy of target detection. However, for battery-powered smart cameras, this will result in extremely high power consumption. For surveillance scenarios using battery-powered cameras, reducing false alarms and missed alarms without increasing power consumption will help improve the performance of smart surveillance cameras. Summary of the invention

[0009] The present invention provides an image-based target detection method to improve the accuracy of target detection.

[0010] A first aspect of the present invention provides an image-based target detection method, the method comprising:

[0011] Get image data,

[0012] Perform motion detection on the acquired image data.

[0013] When a moving target is detected, the three-channel data of the current image frame is converted into two-channel data.

[0014] The motion detection information data obtained by motion detection and the converted two-channel data are input into the trained target detection model for target detection.

[0015] in,

[0016] The motion detection information data enables the target detection model to couple the motion information to the two converted channels in an attention-guided manner during feature extraction, so that the feature values ​​corresponding to the motion edges in the feature map contain semantic features with high information content;

[0017] The target detection model is trained based on motion detection information sample data and two-channel sample data having the same conversion method as the converted two-channel data.

[0018] Preferably, converting the three-channel data of the current image frame into two-channel data includes:

[0019] Merge any two channels of the three channels of the current image frame into composite channel data.

[0020] The step of inputting the motion detection information data obtained by motion detection and the converted two-channel data into the trained target detection model to perform target detection includes:

[0021] Input the motion detection information data, the synthesized channel data, and the unmerged channel data into the trained target detection model for target detection;

[0022] The target detection model is trained based on motion detection information sample data, synthetic channel sample data, and unmerged channel sample data.

[0023] Preferably, the motion detection information data, the synthesized channel data, and the unmerged channel data have the same image size.

[0024] The step of merging any two channel data of the three channel data of the current image frame into synthesized channel data includes:

[0025] For a YUV format image frame, merge the U channel data and the V channel data into composite channel data with the same image size as the Y channel data, or

[0026] For an RGB format image frame, any two of the three RGB channels are merged into composite channel data;

[0027] The same image size means that the size of each channel data is the same in the height direction, and the size of each channel data is the same in the width direction.

[0028] Preferably, the YUV format image frame is in YUV422 format, or YUV420 format, or YUV444 format, or YUV400 format.

[0029] The step of merging the U channel data and the V channel data into synthetic channel data having the same image size as the Y channel data comprises:

[0030] For a YUV422 format image frame, the U channel data in the height direction and the V channel data in the height direction are arranged alternately in the width direction;

[0031] For a YUV420 format image frame, interpolate the U channel data and the V channel data in the height direction respectively, so that the height dimensions of the U channel data and the V channel data are the same as the height dimension of the Y channel data, to obtain interpolated U channel data and interpolated V channel data, and alternately arrange the interpolated U channel data in the height direction and the interpolated Y channel data in the height direction in the width direction;

[0032] For a YUV444 format image frame, add the pixel value of each pixel in the U channel data to the pixel value of the pixel in the V channel data corresponding to the pixel.

[0033] For the YUV400 format image frame, the Y channel data is copied as the composite channel data.

[0034] Preferably, the step of alternately arranging the U channel data in the height direction and the Y channel data in the height direction in the width direction of the image size comprises:

[0035] The i-th row of data in the U channel data is used as the 2i-1th row of data in the synthesized channel data, and the i-th row of data in the V channel data is used as the 2i-th row of data in the synthesized channel data.

[0036] or,

[0037] The i-th row of data in the V channel data is used as the 2i-1-th row of data in the synthesized channel data, and the i-th row of data in the U channel data is used as the 2i-th row of data in the synthesized channel data;

[0038] The interpolation U channel data in the height direction and the interpolation Y channel data in the height direction are alternately arranged in the width direction, including:

[0039] The i-th row of interpolated U channel data is used as the 2i-1th row of synthetic channel data, and the i-th row of interpolated V channel data is used as the 2i-th row of synthetic channel data.

[0040] or,

[0041] The i-th row of data in the interpolated V channel data is used as the 2i-1-th row of data in the synthesized channel data, and the i-th row of data in the interpolated U channel data is used as the 2i-th row of data in the synthesized channel data;

[0042] Wherein, i is a natural number not less than 1.

[0043] Preferably, the performing motion detection on the acquired image data comprises: performing motion detection on the acquired image data using at least one of a frame difference method, an optical flow method, and a background modeling method to obtain motion detection information data including a relationship between previous and next frames,

[0044] The step of merging any two of the three RGB channels into composite channel data includes:

[0045] The pixel value of each pixel point in one channel data is added to the pixel value of the pixel point in another channel data corresponding to the pixel point to obtain the synthesized channel data.

[0046] A second aspect of the present application provides a method for training a target detection model, the method comprising:

[0047] Obtain motion detection information sample data and two-channel sample data, wherein the conversion method of the two-channel sample data is the same as the conversion method of the two-channel data input into the trained target detection model in the inference stage,

[0048] Input the acquired sample data into the target detection model to be trained.

[0049] Determine the loss function value based on the sample detection results output by the target detection model to be trained and the expected detection results.

[0050] Adjust the target detection model to be trained according to the loss function value until the expected result is achieved;

[0051] Among them, the motion detection information sample data enables the target detection model to be trained to couple the motion sample information to the two-channel sample data in an attention-guided manner during feature extraction, so that the sample feature values ​​corresponding to the motion edges in the sample feature map contain semantic features with high information content.

[0052] A third aspect of the present application provides an image-based target detection device, the device comprising:

[0053] An image acquisition module, used to acquire image data,

[0054] A motion detection module is used to perform motion detection on the acquired image data.

[0055] The synthesis module is used to convert the three-channel data of the current image frame into two-channel data when a moving target is detected.

[0056] The target detection module is used to input the motion detection information data obtained by motion detection and the converted two-channel data into the trained target detection model to perform target detection.

[0057] in,

[0058] The target detection model is trained based on motion detection information sample data and two-channel sample data having the same conversion method as the converted two-channel data.

[0059] A fourth aspect of the present application provides a training device for a target detection model, the device comprising:

[0060] The sample data acquisition module is used to acquire motion detection information sample data and two-channel sample data, wherein the conversion method of the two-channel sample data is the same as the conversion method of the two-channel data input by the trained target detection model in the inference stage.

[0061] A training module is used to input the acquired sample data into the target detection model to be trained, determine the loss function value according to the sample detection results output by the target detection model to be trained and the expected detection results, and adjust the target detection model to be trained according to the loss function value until the expected result is achieved; wherein, the motion detection information sample data enables the target detection model to be trained to couple the motion sample information to the two-channel sample data in an attention mechanism-guided manner during feature extraction, so that the sample feature values ​​corresponding to the motion edges in the sample feature map contain semantic features with high information content.

[0062] A fifth aspect of the present application provides an electronic device, which includes the target detection device.

[0063] During the target detection process, the embodiment of the present application not only retains the information of the original image frame, but also introduces motion detection information. While the structure of the target detection model remains unchanged, the background area in the image frame where no motion occurs is filtered out during the target detection process, thereby reducing the difficulty of target detection and reducing the false detection rate and missed detection rate, thereby helping to improve the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of an image-based target detection method according to an embodiment of the present application is provided.

[0065] Figure 2 A flowchart of the target detection model training method.

[0066] Figure 3 A schematic diagram of merging channels of RGB format image frames.

[0067] Figure 4 A schematic diagram of merging channels of YUV422 format image frames.

[0068] Figure 5 A schematic diagram of YUV420 format image frame channel merging.

[0069] Figure 6 A schematic diagram of merging channels of YUV444 format image frames.

[0070] Figure 7 A schematic diagram of YUV400 format image frame channel merging.

[0071] Figure 8 A schematic diagram of a process for performing target detection on a monitoring scene according to an embodiment of the present application.

[0072] Fig. 9 A schematic diagram of inputting motion detection information data, synthesized channel data, and unmerged channel data into a trained target detection model.

[0073] Fig.10 A schematic diagram of an image-based target detection device according to an embodiment of the present application.

[0074] Fig.11 A schematic diagram of a training device for a target detection model according to an embodiment of the present application.

[0075] Fig.12 Another schematic diagram of an image-based target detection device and / or a training device for a target detection model in an embodiment of the present application.

[0076] Fig.13 This is an example of visualization of combining MD image with YUV422 format according to an embodiment of the present application.

[0077] Fig.14 A schematic diagram of the target detection model of the YOLOX architecture. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0079] The applicant found that the open source target detection dataset contains 39,808 images with targets of different sizes in the training set, and 5,204 images with targets in the validation set, for example, pedestrians. For the training set and validation set images, based on the annotated detection frame, the target in the image is moved by a certain amplitude, and the motion detection (MD) image obtained by the frame difference method is combined with the YUV422 format to obtain a visualization image. Fig.13 As shown, the U channel and the V channel are combined into one channel. Fig.13 In the example, since YUV data and RGB data are different, and visualization can only display RGB data, Fig.13 It is pseudo color data. From the visualization results, we can see that due to the addition of MD information, the input YUV image data will produce a very high response in the channel corresponding to MD at the edge of the moving object, such as Fig.13 The gray part in (to meet the requirements of the patent application documents, the pseudo-color highlighted green part is displayed in grayscale) is equivalent to introducing an attention mechanism into the target detection model through MD information. It can be seen that the motion information in the monitoring scene is a very effective guiding information. Combining this information in target detection can effectively improve the accuracy of target detection in the monitoring scene.

[0080] An image-based target detection method provided in an embodiment of the present application uses motion detection information data as guiding information for a target detection task, converts three-channel data contained in a current image frame into two-channel data, and a target detection model performs target detection based on the motion detection information data and the two-channel data.

[0081] See also Figure 1 As shown, Figure 1 This is a flow chart of an image-based target detection method according to an embodiment of the present application, the method comprising:

[0082] Step 101, obtaining image data,

[0083] Step 102: Perform motion detection on the acquired image data.

[0084] As an example, any motion detection algorithm can be used for motion detection. For example, if the frame difference method is used, the obtained motion detection image is the difference between the previous and next frames; for example, if the background modeling method is used, the obtained motion detection image is the difference between the current frame and the background model; for example, if the optical flow method is used, the obtained motion detection image can be the intensity of the optical flow, thereby obtaining motion detection information, which contains the relationship between the previous and next frames, and is a very good guide information for target detection tasks, which is equivalent to filtering out the background area without motion during the target detection process. In this way, the difficulty of target detection can be greatly reduced, thereby reducing the false detection rate and missed detection rate.

[0085] Regardless of the method used to obtain motion detection information, the information is extracted from the image time domain. The motion detection information contains the motion features of the corresponding position of the image and is used to form an attention mechanism in the target detection model.

[0086] It should be understood that the two frames refer to the time difference between the two frames being no greater than a set time threshold, for example, the time difference is less than or equal to 100ms to 200ms.

[0087] Step 103, when a moving target is detected, convert the three-channel data of the current image frame into two-channel data.

[0088] As an example, the conversion method is selected according to the frame format of the current image frame so that there is no information loss between the channel data before the conversion and the channel data after the conversion. In this way, without changing the model structure of the target detection model, the motion detection image channel data can be increased and the information amount of the original image input of the target detection model can be retained.

[0089] Step 104: input the motion detection information data obtained by motion detection and the converted two-channel data into the trained target detection model to perform target detection.

[0090] As an example, the motion detection information data, the synthesized channel data, and the unmerged channel data have the same image size, that is, the size of each channel data in the height direction is the same, and the size of each channel data in the width direction is the same. In this way, there is no need to change the structure of the target detection model;

[0091] in,

[0092] The motion detection information data enables the target detection model to couple the motion information to the two converted channels in an attention-guided manner during feature extraction, so that the feature values ​​corresponding to the motion edges in the feature map contain semantic features with high information content;

[0093] The target detection model is trained based on motion detection information sample data and two-channel sample data having the same conversion method as the converted two-channel data.

[0094] The embodiment of the present application not only retains the information of the current image frame itself, but also utilizes motion detection information. It increases the amount of input information without changing the structure of the target detection model. Moreover, the added motion detection information is equivalent to establishing an attention mechanism in the target detection model, so that the current detection model forms a more obvious difference in the feature extraction of the current frame image, thereby effectively improving the robustness of the target detection model and reducing the false alarm rate and the missed alarm rate.

[0095] To facilitate understanding of the embodiments of the present application, the image-based target detection method of the embodiments of the present application is described below in combination with different image frame formats. It should be understood that the embodiments of the present application are not limited to a certain image frame format, and any image frame format can be applied to the image-based target detection method of the embodiments of the present application.

[0096] See also Figure 2 As shown, Figure 2 The following is a flowchart of a method for training a target detection model. The training method includes:

[0097] Step 201, obtaining sample data for training the target detection model to be trained,

[0098] As an example, sample sequence images or sample videos are collected, and sample objects in the sample images are annotated.

[0099] Convert the format of the collected sample image to the image frame format required by the trained target detection model in the inference phase, for example, convert RGB data into YUV data.

[0100] A motion detection algorithm is used to perform motion detection on sample images of previous and next frames to generate motion detection sample images. For example, any motion detection algorithm can be used for motion detection, including but not limited to at least one of a frame difference method, a background modeling method, an optical flow method, etc. Preferably, motion detection is performed by using a detection method that uses the motion detection information data that needs to be input in the inference stage of a trained target detection model.

[0101] As an example, in combination with the image frame format, the channel sample data of the sample image after the conversion format is converted into two-channel sample data. For example:

[0102] If the image frame format is RGB format, any two channels of the RGB three-channel sample data are merged into the composite channel sample data. Specifically, Figure 3As shown, the pixel value of each pixel point in the sample data of one channel is added to the pixel value of the pixel point in the sample data of another channel corresponding to the pixel point to obtain the synthesized channel sample data.

[0103] If the image frame format is YUV format, any two channels of the YUV three-channel sample data are merged into the composite channel sample data; in view of the fact that Y in the YUV format is used to represent grayscale information, and U and V represent color information, in order to better retain the image feature information, the U channel sample data and the V channel sample data are merged into the composite channel sample data. Specifically,

[0104] When the YUV format is YUV422 format, such as Figure 4 As shown, since the width of the U channel and the V channel are both half of the width W of the Y channel, that is, W / 2, and the height of the U channel and the V channel are the same as the height of the Y channel, the U channel sample data in the height direction and the V channel sample data in the height direction are arranged alternately in the width direction, for example,

[0105] The i-th row of data in the U channel sample data is used as the 2i-1th row of data in the synthesized channel sample data, and the i-th row of data in the V channel sample data is used as the 2i-th row of data in the synthesized channel sample data, or the i-th row of data in the V channel sample data is used as the 2i-1th row of data in the synthesized channel sample data, and the i-th row of data in the U channel sample data is used as the 2i-th row of data in the synthesized channel sample data;

[0106] When the YUV format is YUV420 format, such as Figure 5 As shown, since the widths of the U channel and the V channel are both half of the width W of the Y channel, that is, W / 2, and the heights of the U channel and the V channel are both half of the height H of the Y channel, that is, H / 2, the U channel sample data and the V channel sample data are interpolated in the height direction respectively, so that the height dimensions of the U channel sample data and the channel sample data are the same as the height dimension of the Y channel sample data, and the interpolated U channel sample data and the interpolated V channel sample data are obtained. The interpolated U channel sample data in the height direction and the interpolated Y channel sample data in the height direction are alternately arranged in the width direction, for example,

[0107] Using the i-th row of data in the interpolated U channel sample data as the 2i-1th row of data in the synthesized channel sample data, and using the i-th row of data in the interpolated V channel sample data as the 2i-th row of data in the synthesized channel sample data, or using the i-th row of data in the interpolated V channel sample data as the 2i-1th row of data in the synthesized channel sample data, and using the i-th row of data in the interpolated U channel sample data as the 2i-th row of data in the synthesized channel sample data;

[0108] Wherein, i is a natural number not less than 1.

[0109] When the YUV format is YUV444 format, such as Figure 6 As shown, since the heights and widths of the U channel, V channel, and Y channel are the same, the pixel value of each pixel point in the U channel sample data is added to the pixel value of the pixel point in the V channel sample data corresponding to the pixel point to obtain the synthesized channel sample data;

[0110] When the YUV format is YUV400 format, such as Figure 7 As shown, since there is no U channel and V channel information data, the Y channel sample data is copied as the synthetic channel sample data.

[0111] It should be understood that the YUV format is not limited to the formats listed above. When it is in other formats, the channel data can be merged according to the height and width relationship between the channels.

[0112] Step 202: input the motion detection sample image data, the synthesized channel sample data, and the unmerged channel sample data into the target detection model to be trained.

[0113] For example, for any sample image frame, the synthesized channel sample data of the sample image frame, the merged channel sample data, and the running detection sample image data obtained with the help of the previous and next image frames of the sample image frame are input into the target detection model to be trained.

[0114] The motion detection information sample data enables the target detection model to be trained to couple the motion sample information to the two-channel sample data in an attention-guided manner during feature extraction, so that the sample feature values ​​corresponding to the motion edges in the sample feature graph contain semantic features with high information content, which is beneficial to improving training efficiency and effects.

[0115] Step 203, calculating the loss function value between the sample target detection result output by the target detection model to be trained and the labeled sample target,

[0116] Step 204, adjust the model parameters of the target detection model to be trained according to the loss function value, and repeatedly perform steps 202 to 204 until the target detection model to be trained is as expected.

[0117] Thus, the trained target detection model is obtained.

[0118] See also Figure 8 As shown, Figure 8 The present invention is a flowchart of an embodiment of the present invention for detecting a target in a monitoring scene. The target detection method includes: on the camera device side,

[0119] Step 801, obtaining image data of the monitored scene,

[0120] Step 802: Perform motion detection on the acquired image data.

[0121] As an example, at least one of a frame difference method, an optical flow method, and a background modeling method may be used to perform motion detection on the acquired image data to obtain motion detection information data including the relationship between previous and next frames.

[0122] Step 803: determine whether there is a moving target based on the motion detection information.

[0123] If there is a moving target, then execute step 804.

[0124] Otherwise, return to step 801,

[0125] Step 804: merge two channel data of the three channel data in the current image frame into one channel data to obtain synthesized channel data.

[0126] As an example,

[0127] If the image frame format is RGB format, any two channels of the RGB three-channel data are merged into synthetic channel data. Specifically, Figure 3 As shown, the pixel value of each pixel point in one channel data is added to the pixel value of the pixel point in another channel data corresponding to the pixel point to obtain the synthesized channel data.

[0128] If the image frame format is YUV format, any two channels of the YUV three-channel sample data are merged into the composite channel sample data; in view of the fact that Y in the YUV format is used to represent grayscale information, and U and V represent color information, in order to better retain the image feature information, the U channel sample data and the V channel sample data are merged into the composite channel sample data. Specifically,

[0129] When the YUV format is YUV422 format, such as Figure 4 As shown, since the width of the U channel and the V channel are both half of the width W of the Y channel, that is, W / 2, and the height of the U channel and the V channel are the same as the height of the Y channel, the U channel data in the height direction and the V channel data in the height direction are alternately arranged in the width direction, for example,

[0130] The i-th row of data in the U channel data is used as the 2i-1th row of data in the synthesized channel data, and the i-th row of data in the V channel data is used as the 2i-th row of data in the synthesized channel data, or the i-th row of data in the V channel data is used as the 2i-1th row of data in the synthesized channel data, and the i-th row of data in the U channel data is used as the 2i-th row of data in the synthesized channel data;

[0131] When the YUV format is YUV420 format, such as Figure 5 As shown, since the widths of the U channel and the V channel are both half of the width W of the Y channel, that is, W / 2, and the heights of the U channel and the V channel are both half of the height H of the Y channel, that is, H / 2, the U channel data and the V channel data are interpolated in the height direction respectively, so that the height dimensions of the U channel data and the V channel data are the same as the height dimension of the Y channel data, and the interpolated U channel data and the interpolated V channel data are obtained. The interpolated U channel data in the height direction and the interpolated Y channel data in the height direction are alternately arranged in the width direction. For example,

[0132] Using the V ith row of interpolated U channel data as the 2i-1th row of data in the synthesized channel data, and using the ith row of interpolated V channel data as the 2ith row of data in the synthesized channel data, or using the ith row of interpolated V channel data as the 2i-1th row of data in the synthesized channel data, and using the ith row of interpolated U channel data as the 2ith row of data in the synthesized channel data;

[0133] Wherein, i is a natural number not less than 1.

[0134] When the YUV format is YUV444 format, such as Figure 6 As shown, since the heights and widths of the U channel, V channel, and Y channel are the same, the pixel value of each pixel in the U channel data is added to the pixel value of the pixel in the V channel data corresponding to the pixel to obtain the synthesized channel data;

[0135] When the YUV format is YUV400 format, such as Figure 7 As shown, since there is no U channel and V channel information data, the Y channel data is copied as the composite channel data.

[0136] It should be understood that the YUV format is not limited to the above-listed formats. When it is in other formats, the channel data can be merged according to the height and width relationship between the channels. Since the detection capability of the target detection model is image-level detection rather than pixel-level detection, the specific way to merge the channel data can be determined through experiments.

[0137] Step 805 , input the motion detection information data, the synthesized channel data, and the unmerged channel data into the trained target detection model to perform target detection.

[0138] See also Fig. 9 As shown, Fig. 9A schematic diagram of inputting motion detection information data, synthetic channel data, and unmerged channel data into a trained target detection model. The image frame format is YUV422, and the motion detection information data, synthetic channel data, and unmerged channel data have the same image size. This embodiment uses the inter-frame information generated by motion detection during the target detection process. Compared with the detection method that directly uses the previous and next frame images for target detection, it reduces power consumption, thereby saving power consumption. Compared with the detection method that directly uses the three-channel data of the image frame for target detection, it reduces the false detection rate and missed detection rate, and improves the accuracy of target detection without changing the structure of the target detection model itself.

[0139] See also Fig.10 As shown, Fig.10 This is a schematic diagram of an image-based target detection device according to an embodiment of the present application, the device comprising:

[0140] An image acquisition module, used to acquire image data,

[0141] A motion detection module is used to perform motion detection on the acquired image data.

[0142] The synthesis module is used to convert the three-channel data of the current image frame into two-channel data when a moving target is detected.

[0143] A target detection module, used for performing target detection based on the motion detection information data obtained by motion detection and the converted two-channel data;

[0144] The motion detection information data enables the target detection model to couple the motion information to the two converted channels in an attention-guided manner during feature extraction, so that the feature values ​​corresponding to the motion edges in the feature map contain semantic features with high information content.

[0145] See also Fig.11 As shown, Fig.11 A schematic diagram of a training device for a target detection model according to an embodiment of the present application. The device includes:

[0146] The sample data acquisition module is used to acquire motion detection information sample data and two-channel sample data.

[0147] The training module is used to input the acquired sample data into the target detection model to be trained, determine the loss function value according to the sample detection results output by the target detection model to be trained and the expected detection results, and adjust the target detection model to be trained according to the loss function value until the expected

[0148] Among them, the motion detection information sample data enables the target detection model to be trained to couple the motion sample information to the two-channel sample data in an attention-guided manner during feature extraction, so that the sample feature values ​​corresponding to the motion edges in the sample feature map contain semantic features with high information content.

[0149] As an example, the sample data acquisition module includes:

[0150] The sample image data acquisition submodule is used to acquire sample image frame data.

[0151] The motion detection submodule is used to perform motion detection on the collected sample image frames.

[0152] The synthesis submodule is used to convert the three-channel data of the sample image frame into two-channel data.

[0153] See also Fig.12 As shown, Fig.12 Another schematic diagram of an image-based target detection device and / or a target detection model training device according to an embodiment of the present application. The device includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the image-based target detection method and / or the target detection model training method according to an embodiment of the present application.

[0154] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0155] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0156] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the training method for the target detection model for low-light images and / or the steps of the target detection method based on low-light images.

[0157] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] The following is an explanation using the target detection model of the YOLOX architecture as an example. It should be understood that the target detection model of the present application is not limited to this architecture, and the target detection model of any neural network structure can be applied to the present application.

[0159] See also Fig.14 As shown, Fig.14 A schematic diagram of the target detection model of the YOLOX architecture. The model includes a first feature extraction module that samples the input data at an 8-fold rate, a second feature extraction module that samples at a 16-fold rate, and a third feature extraction module that samples at a 32-fold rate. The output feature data of the first feature extraction module is sent to the first detection head, the output feature data of the second feature extraction module is sent to the second detection head, and the output feature data of the third feature extraction module is sent to the third detection head.

[0160] The motion detection information image with a height of H and a width of W, the Y channel image, and the synthetic channel image are input into the first feature extraction module, and after being sampled at a rate of 8, a first feature map with a height of H / 8 and a width of W / 8 and including a first scale detection result C1 is obtained. The first feature map is recognized by the first detection head to obtain a first detection result; wherein, under the guidance of the attention mechanism of the motion detection information image, the motion information is coupled to the residual channel, so that the feature value corresponding to the motion edge in the first feature map contains semantic features with high information content,

[0161] The first feature map is input to the second feature extraction module, and after being sampled at a rate of 16, a second feature map with a height of H / 16 and a width of W / 16, and including a second scale detection result C2, is obtained. The second feature map is identified by the second detection head to obtain a second detection result; wherein, under the guidance of the attention mechanism of the motion detection information image, the motion information is coupled to the residual channel, so that the feature value corresponding to the motion edge in the second feature map contains semantic features with high information content,

[0162] The second feature map is input to the third feature extraction module, and after sampling at a rate of 32, a third feature map with a height of H / 32, a width of W / 32, and including a third scale detection result C3 is obtained. The third feature map is recognized by the third detection head to obtain a third detection result, wherein, under the guidance of the attention mechanism of the motion detection information image, the motion information is coupled to the remaining two channels, so that the feature value corresponding to the motion edge in the third feature map contains semantic features with high information content.

[0163] The detection results output by each detection head include: detection box regression value, category score and target score, where Cls is the number of target categories.

[0164] In order to verify the effectiveness of the present invention, two experiments were conducted. The first experiment was a benchmark experiment, that is, the original RGB data was used as the input of the model for training, and the second experiment was to use the technical solution of the present invention. Both experiments used the same data augmentation scheme, the same training strategy, and the same target detection label. After the training was completed, verification was performed on the same validation set. The input resolution used for verification was 416*416, and the evaluation indicators used for verification were mAP, mAP20 and mAP50. The final indicator results are shown in the table below. From the results in the table, it can be seen that the solution proposed by the present invention has brought significant improvements in detection effect.

[0165] enter Input size mAP mAP 50 mAP 20 RGB 416*416*3 35.6 67.0 76.7 Y, MD, UV synthesis 416*416*3 40.2 72.2 80.6

[0166] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image-based target detection method, characterized in that: The method includes: Get image data, Perform motion detection on the acquired image data. When a moving target is detected, the three-channel data of the current image frame is converted into two-channel data. The motion detection information data obtained by motion detection and the converted two-channel data are input into the trained target detection model for target detection. in, The motion detection information data enables the target detection model to couple the motion information to the two converted channels in an attention-guided manner during feature extraction, so that the feature values ​​corresponding to the motion edges in the feature map contain semantic features with high information content; The target detection model is trained based on motion detection information sample data and two-channel sample data having the same conversion method as the converted two-channel data.

2. The target detection method according to claim 1, characterized in that: The converting the three-channel data of the current image frame into two-channel data includes: Merge any two channels of the three channels of the current image frame into composite channel data. The step of inputting the motion detection information data obtained by motion detection and the converted two-channel data into the trained target detection model to perform target detection includes: Input the motion detection information data, the synthesized channel data, and the unmerged channel data into the trained target detection model for target detection; The target detection model is trained based on motion detection information sample data, synthetic channel sample data, and unmerged channel sample data.

3. The target detection method according to claim 2, characterized in that: The motion detection information data, the synthesized channel data, and the unmerged channel data have the same image size, The step of merging any two channel data of the three channel data of the current image frame into synthesized channel data includes: For a YUV format image frame, merge the U channel data and the V channel data into composite channel data with the same image size as the Y channel data, or For an RGB format image frame, any two of the three RGB channels are merged into composite channel data; The same image size means that the size of each channel data is the same in the height direction, and the size of each channel data is the same in the width direction.

4. The target detection method according to claim 3, characterized in that: The YUV format image frame is in YUV422 format, or YUV420 format, or YUV444 format, or YUV400 format. The step of merging the U channel data and the V channel data into synthetic channel data having the same image size as the Y channel data comprises: For a YUV422 format image frame, the U channel data in the height direction and the V channel data in the height direction are arranged alternately in the width direction; For a YUV420 format image frame, interpolate the U channel data and the V channel data in the height direction respectively, so that the height dimensions of the U channel data and the V channel data are the same as the height dimension of the Y channel data, to obtain interpolated U channel data and interpolated V channel data, and alternately arrange the interpolated U channel data in the height direction and the interpolated Y channel data in the height direction in the width direction; For a YUV444 format image frame, add the pixel value of each pixel in the U channel data to the pixel value of the pixel in the V channel data corresponding to the pixel. For the YUV400 format image frame, the Y channel data is copied as the composite channel data.

5. The target detection method according to claim 4, characterized in that: The step of alternately arranging the U channel data in the height direction and the Y channel data in the height direction in the width direction of the image size comprises: The i-th row of data in the U channel data is used as the 2i-1th row of data in the synthesized channel data, and the i-th row of data in the V channel data is used as the 2i-th row of data in the synthesized channel data. or, The i-th row of data in the V channel data is used as the 2i-1-th row of data in the synthesized channel data, and the i-th row of data in the U channel data is used as the 2i-th row of data in the synthesized channel data; The interpolation U channel data in the height direction and the interpolation Y channel data in the height direction are alternately arranged in the width direction, including: The i-th row of interpolated U channel data is used as the 2i-1th row of synthetic channel data, and the i-th row of interpolated V channel data is used as the 2i-th row of synthetic channel data. or, The i-th row of data in the interpolated V channel data is used as the 2i-1-th row of data in the synthesized channel data, and the i-th row of data in the interpolated U channel data is used as the 2i-th row of data in the synthesized channel data; Wherein, i is a natural number not less than 1.

6. The target detection method according to claim 3, characterized in that: The motion detection of the acquired image data includes: using at least one of a frame difference method, an optical flow method, and a background modeling method to perform motion detection on the acquired image data to obtain motion detection information data including the relationship between the previous and next frames, The step of merging any two of the three RGB channels into composite channel data includes: The pixel value of each pixel point in one channel data is added to the pixel value of the pixel point in another channel data corresponding to the pixel point to obtain the synthesized channel data.

7. A method for training a target detection model, characterized in that: The method includes: Obtain motion detection information sample data and two-channel sample data, wherein the conversion method of the two-channel sample data is the same as the conversion method of the two-channel data input into the trained target detection model in the inference stage, Input the acquired sample data into the target detection model to be trained. Determine the loss function value based on the sample detection results output by the target detection model to be trained and the expected detection results. Adjust the target detection model to be trained according to the loss function value until the expected result is achieved; Among them, the motion detection information sample data enables the target detection model to be trained to couple the motion sample information to the two-channel sample data in an attention-guided manner during feature extraction, so that the sample feature values ​​corresponding to the motion edges in the sample feature map contain semantic features with high information content.

8. An image-based target detection device, characterized in that: The device includes: An image acquisition module, used to acquire image data, A motion detection module is used to perform motion detection on the acquired image data. The synthesis module is used to convert the three-channel data of the current image frame into two-channel data when a moving target is detected. The target detection module is used to input the motion detection information data obtained by motion detection and the converted two-channel data into the trained target detection model to perform target detection. in, The motion detection information data enables the target detection model to couple the motion information to the two converted channels in an attention-guided manner during feature extraction, so that the feature values ​​corresponding to the motion edges in the feature map contain semantic features with high information content; The target detection model is trained based on motion detection information sample data and two-channel sample data having the same conversion method as the converted two-channel data.

9. A training device for a target detection model, characterized in that: The device includes: The sample data acquisition module is used to acquire motion detection information sample data and two-channel sample data, wherein the conversion method of the two-channel sample data is the same as the conversion method of the two-channel data input by the trained target detection model in the inference stage. The training module is used to input the acquired sample data into the target detection model to be trained, determine the loss function value according to the sample detection results output by the target detection model to be trained and the expected detection results, and adjust the target detection model to be trained according to the loss function value until the expected result is achieved; Among them, the motion detection information sample data enables the target detection model to be trained to couple the motion sample information to the two-channel sample data in an attention-guided manner during feature extraction, so that the sample feature values ​​corresponding to the motion edges in the sample feature map contain semantic features with high information content.

10. An electronic device, characterized in that: The electronic device comprises the object detection device as claimed in claim 8.

Images