Target object detection method, device and computer storage medium
By combining the dynamic and static characteristics of video frames in flame detection, the problem of high flame detection error detection in the prior art is solved, more accurate flame recognition is achieved, and efficient detection processing is maintained.
Patent Information
- Application Number
- CN202210460633.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-28
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2042-04-28
AI Technical Summary
The existing deep learning flame detection methods only use the static characteristics of the picture, making it difficult to distinguish similar objects of flames, such as alarm lights, reflective clothes or flame patterns on pedestrian clothes, resulting in a high false detection rate.
By obtaining multiple target image frames in the video to be detected, extracting and splicing dynamic and static features of the target object, and performing classification predictions to obtain the detection results of the target object, improving the accuracy of the detection.
The error judgment rate of flame detection is reduced, the accuracy of detection results is improved, and the impact on the model calculation amount is small, and the detection and processing efficiency is high.
Smart Images
Figure CN114898256B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of image recognition technology, and in particular to a target object detection method, device and computer storage medium. Background Art
[0002] Conventional flame identification usually requires the use of sensors to detect changes in physical and chemical parameters such as particle concentration and temperature caused by flame combustion. Although traditional detection methods and corresponding commercial sensors have been widely used, there is still room for innovation in the intelligence, coverage and cost of the methods.
[0003] Fireworks detection based on deep learning has the advantages of high intelligence, wide coverage, low cost and strong anti-interference. In practical applications, high-precision recognition and detection of flame morphology can be achieved using ordinary cameras, which has important application value in special working conditions such as outdoor.
[0004] However, the currently commonly used deep learning flame detection method only uses the static features of the image and does not use the motion characteristics of the flame, which makes it difficult to distinguish many flame-like objects, such as warning lights, reflective clothing, or flame patterns printed on pedestrians' clothes. Summary of the invention
[0005] In view of the above problems, the present application provides a target object detection technical solution, which can at least partially solve the various problems in the prior art.
[0006] The first aspect of the present application provides a target object detection method, comprising: obtaining multiple target image frames in a video to be detected; determining a current image frame in each target image frame, detecting multiple target image frames including the current image frame, and obtaining dynamic features of the target object; splicing the dynamic features of the target object and the static features extracted from the current image frame to obtain spliced features; performing classification prediction on the spliced features to obtain a target object detection result of the current image frame.
[0007] The second aspect of the present application provides a target object detection device, including: an acquisition module, used to acquire multiple target image frames in a video to be detected; a feature analysis module, used to determine the current image frame in each target image frame, detect multiple target image frames including the current image frame, and obtain dynamic features of the target object; a splicing module, used to splice the dynamic features of the target object and the static features extracted from the current image frame to obtain splicing features; a detection module, used to perform classification prediction on the splicing features to obtain the detection result of the target object.
[0008] The third aspect of the present application provides a computer storage medium, in which instructions for executing the steps in the method described in the first aspect are stored.
[0009] In summary, the target object detection method, device and computer storage medium provided by various aspects of the present application obtain the dynamic features of the target object by detecting the target object in multiple target image frames including the current image frame, and obtain the detection result of the target object by splicing the dynamic features of the target object and the static features of the current image frame. Accordingly, the present application integrates the dynamic features of the target object without losing the static features of the image, which can improve the accuracy of the fireworks detection results, reduce the misjudgment rate, and has little impact on the model calculation amount, and has a high detection processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0011] Figure 1 This is a processing flow chart of a target object detection method according to an exemplary embodiment of the present application.
[0012] Figure 2 This is a processing flow chart of a target object detection method according to another exemplary embodiment of the present application.
[0013] Figure 3 This is a processing flow chart of a target object detection method according to another exemplary embodiment of the present application.
[0014] Figure 4 A schematic diagram of an embodiment of feature stitching performed in this application.
[0015] Figure 5 This is a processing flow chart of a target object detection method according to another exemplary embodiment of the present application.
[0016] Figures 6 to 8 Schematic diagram of flame detection at different processing stages of the target object detection method of the present application.
[0017] Fig. 9 It is a structural block diagram of a target object detection device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0018] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0019] The currently commonly used deep learning flame detection method only uses the static features of the image, which makes it difficult to distinguish similar objects of flames, such as alarm lights, reflective clothing, or flame patterns printed on pedestrians' clothes, resulting in a high flame false detection rate. In view of this, each embodiment of the present application provides a target object detection solution that can effectively improve the problem of high flame false detection rate.
[0020] The embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.
[0021] Figure 1 This is the processing flow of the target object detection method of the exemplary embodiment of the present application. As shown in the figure, this embodiment mainly includes the following steps:
[0022] Step S102, obtaining multiple target image frames in the video to be detected.
[0023] Optionally, based on a preset frame interval, each original image frame in the video to be detected may be extracted in sequence, and normalization processing may be performed on the pixel value of each pixel point in each original image frame to obtain a target image frame of each original image frame.
[0024] Specifically, the pixel value of each pixel in each original image frame may be normalized so that the pixel value of each pixel is between 0 and 1.
[0025] Optionally, after the step of sequentially extracting each original image frame in the video to be detected, the image size of each original image frame may be adjusted based on a preset image size, and then normalization processing is performed on each original image frame after the size adjustment.
[0026] Optionally, the preset image size can be set to 640*384, but is not limited thereto. It can also be set to other sizes according to actual detection requirements, and this application does not impose any restrictions on this.
[0027] Step S104: determine a current image frame among the target image frames, detect multiple target image frames including the current image frame, and obtain dynamic features of the target object.
[0028] Alternatively, the target object may include a flame.
[0029] Optionally, according to the determined current image frame, a plurality of previous image frames of the current image frame may be acquired from the target image frame, and the current image frame and the previous image frames may be detected to generate a motion estimation vector matrix of the target object.
[0030] In this embodiment, each matrix element in the motion estimation vector matrix is used to identify the motion estimation value of each pixel in the current image frame, which can be used to represent the dynamic characteristics of the target object.
[0031] Step S106, splicing the dynamic features of the target object and the static features extracted from the current image frame to obtain splicing features.
[0032] Optionally, a first stitching may be performed on the dynamic features of the target object and the static features extracted from the current image frame to obtain a primary stitching image, and then a second stitching may be performed on the dynamic features of the target object and the static features extracted from the primary stitching image to obtain a secondary stitching image.
[0033] Step S108, performing classification prediction on the splicing features to obtain the target object detection result of the current image frame.
[0034] Optionally, classification prediction may be performed on the splicing features to obtain a detection result of whether the target object is included in the current image frame.
[0035] Optionally, classification prediction may be performed for each pixel in the secondary splicing image to determine a target area in the secondary splicing image containing the target object. If a connected domain is identified in the target area, a detection result of flame presence in the current video frame is obtained.
[0036] Optionally, if the detection result is that there is flame in the current video frame, the flame area in the current image frame may be further output.
[0037] In summary, the target object detection method provided in the embodiment of the present application obtains the detection result of the target object by analyzing the dynamic characteristics of the target object and splicing the dynamic characteristics and static characteristics of the target object. By means of the above technical means, the false detection rate of flames can be effectively reduced and the accuracy of the detection results can be improved.
[0038] Figure 2 The processing flow of the target object detection method of another exemplary embodiment of the present application is shown. This embodiment is a specific implementation of the above step S104. As shown in the figure, this embodiment mainly includes the following processing steps:
[0039] Step S202: determine a target image frame as the current image frame.
[0040] Step S204: extracting a plurality of previous image frames of the current image frame from each target image frame based on the current image frame.
[0041] In this embodiment, each previous image frame is a target image frame having a time stamp smaller than that of the current image frame, and each target image frame constituting each previous image frame and the current time frame are sequentially continuous.
[0042] Optionally, the number of the extracted previous image frames may be between 4 and 19.
[0043] Preferably, 5 previous image frames may be extracted.
[0044] Step S206: Detect the current image frame and each previous image frame to obtain a motion estimation vector matrix of the target object.
[0045] In this embodiment, each matrix element in the motion estimation vector matrix may be used to identify the motion estimation value of each pixel in the current image frame.
[0046] Optionally, the current image frame and each previous image frame may be detected using a light streamer method or a frame difference method to obtain a motion estimation vector matrix of the target object.
[0047] In one embodiment, when the light streamer method is used to detect the current image frame and each previous image frame, the grayscale instantaneous change rate value of each pixel in the current image frame can be obtained.
[0048] In another embodiment, when the frame difference method is used to detect the current image frame and each previous image frame, the grayscale difference value of each pixel in the current image frame can be obtained.
[0049] In summary, the embodiments of the present application detect the current image frame and multiple previous image frames of the current image frame to obtain the dynamic features of the target object, thereby helping to improve the accuracy of the target object detection result.
[0050] Figure 3 The processing flow of the target object detection method of another exemplary embodiment of the present application is shown. This embodiment mainly shows the specific implementation scheme of the above step S106. As shown in the figure, this embodiment mainly includes the following steps:
[0051] Step S302, performing feature extraction on the current image frame according to the image channel of the current image frame to obtain a static feature map of the current image frame.
[0052] For example, assuming that the target image frame is a three-channel image, the length and width of the target image frame are w and h respectively, and feature extraction is performed on the current image frame to obtain a static feature map of the current image frame with a dimension of w×h×3.
[0053] Step S304: based on the pixels at the same position, the motion estimation vector matrix of the target object and the static feature map of the current image frame are spliced to obtain a spliced image.
[0054] In this embodiment, according to the motion estimation value (such as grayscale instantaneous change rate value or grayscale difference value) of each pixel point in the motion estimation vector matrix and the three-channel value of each pixel point in the static feature map, the motion estimation value and the three-channel value of the pixel point at the same position are spliced to obtain a primary splicing image with a dimension of w×h×4(3+1) (refer to Figure 4 ).
[0055] Step S306: Based on the preset feature dimension, feature extraction is performed on the primary spliced image to obtain a primary feature map of the primary spliced image.
[0056] Optionally, a preset flame segmentation algorithm may be used to perform feature extraction on a primary splicing graph based on a preset feature dimension to obtain a primary feature graph.
[0057] Optionally, the preset flame segmentation algorithm may include but is not limited to Deeplabv3, Unet algorithm, etc.
[0058] Specifically, the trained flame semantic segmentation model can be used to perform feature extraction on a spliced image based on a preset feature dimension c to obtain a feature map with a dimension of w×h×c (refer to Figure 4 ).
[0059] Step S308: based on the pixels at the same position, the motion estimation vector matrix of the target object and the primary feature map are spliced to obtain a secondary spliced map.
[0060] In this embodiment, according to the motion estimation value (e.g., grayscale instantaneous change rate value or grayscale difference value) of each pixel in the motion estimation vector matrix and the c-dimensional feature value of each pixel in the primary feature map, the motion estimation value and the c-dimensional feature value of the pixel at the same position are spliced to obtain a secondary spliced map with a dimension of w×h×(c+1) (refer to Figure 4 ).
[0061] In summary, this embodiment can improve the accuracy of subsequent target object detection results by fusing the dynamic features and static features of the target object, and has little impact on the model calculation amount, thereby ensuring the processing efficiency of the model.
[0062] Figure 5 The processing flow of the target object detection method of another exemplary embodiment of the present application is shown. This embodiment is a specific implementation of the above step S108. As shown in the figure, this embodiment mainly includes the following steps:
[0063] Step S502 , performing classification prediction on each pixel in the secondary spliced image to determine a target area in the secondary spliced image that contains the target object.
[0064] Optionally, each pixel point in the secondary splicing image can be input into the semantic segmentation network pixel classifier for classification prediction to divide the flame area and the non-flame area, and obtain Figure 6 The classification prediction result graph of the secondary splicing graph is shown.
[0065] Step S504: Perform morphological processing on the target area using a preset morphological processing algorithm.
[0066] Optionally, the preset morphological processing algorithm includes at least one of an opening operation and a closing operation.
[0067] Specifically, a preset morphological processing algorithm can be used to perform morphological processing such as removing holes and burrs on the target area in the secondary splicing image to obtain Figure 7 The morphological processing results of the secondary mosaic image are shown.
[0068] Step S506, determine whether there is a connected domain in the target area, if yes, proceed to step S508, if not, proceed to step S510.
[0069] Step S508: obtaining a detection result of the target object based on the identified connected domain.
[0070] Optionally, edge detection may be performed on the connected domain based on a preset edge detection algorithm to determine the edge contour of the connected domain.
[0071] Optionally, the preset edge detection algorithm may include at least one of a Canny algorithm and a Sobel algorithm.
[0072] Optionally, the determined edge contour of the connected domain may be output as a detection result of the target object.
[0073] Specifically, by performing edge detection on the connected domain and traversing the outer boundary of the connected domain, the contour curve of the flame can be obtained.
[0074] Optionally, rectangle fitting may be performed based on the edge contour of the connected domain to obtain a detection result of the target object.
[0075] Specifically, the minimum circumscribed rectangle fitting can be performed on the contour curve of the flame to obtain the detection frame of the target object, and the detection frame is output as the detection result of the target object (refer to Figure 8 ).
[0076] Step S510, obtaining a detection result that no target object exists.
[0077] Specifically, if a connected domain cannot be identified from the target area, a detection result indicating that no flame exists in the secondary spliced image (current image frame) is obtained.
[0078] In summary, the target object detection result provided by the embodiment of the present application can not only identify the detection result of whether there is a flame in the current image frame by identifying the connected domain in the target area, but also accurately detect the flame area in the current image frame.
[0079] Fig. 9 The structure block diagram of the target object detection device of the exemplary embodiment of the present application is shown. As shown in the figure, the target object detection device 900 of the present embodiment mainly includes: an acquisition module 902, a feature analysis module 904, a splicing module 906, and a detection module 908.
[0080] The acquisition module 902 is used to acquire multiple target image frames in the video to be detected.
[0081] The feature analysis module 904 is used to determine the current image frame among the target image frames, detect multiple target image frames including the current image frame, and obtain dynamic features of the target object.
[0082] The stitching module 906 is used to stitch the dynamic features of the target object and the static features extracted from the current image frame to obtain stitching features.
[0083] The detection module 908 is used to perform classification prediction on the splicing features to obtain the detection result of the target object.
[0084] Optionally, the target object comprises a flame.
[0085] Optionally, the acquisition module 902 is further used to: extract each original image frame in the video to be detected in sequence based on a preset frame interval; perform normalization processing on the pixel value of each pixel point in each original image frame to obtain a target image frame of each original image frame.
[0086] Optionally, the acquisition module 902 is further configured to: adjust the image size of each original image frame based on a preset image size.
[0087] Optionally, the feature analysis module 904 is also used to: determine a target image frame as a current image frame; based on the current image frame, extract multiple previous image frames of the current image frame from each target image frame; detect the current image frame and each previous image frame to obtain a motion estimation vector matrix of the target object; wherein each previous image frame is a target image frame whose timestamp is less than the timestamp of the current image frame, and each target image frame constituting each previous image frame and the current time frame are continuous in sequence; wherein the number of extracted previous image frames is between 4 and 19; wherein each matrix element in the motion estimation vector matrix is used to identify the motion estimation value of each pixel point in the current image frame.
[0088] Optionally, the feature analysis module 904 is also used to: use the light streaming method or the frame difference method to detect the current image frame and each previous image frame to obtain the motion estimation vector matrix of the target object; wherein each matrix element in the motion estimation vector matrix is used to identify the grayscale instantaneous change rate value or grayscale difference value of each pixel point in the current image frame.
[0089] Optionally, the stitching module 906 is further used to: perform feature extraction on the current image frame according to the image channel of the current image frame to obtain a static feature map of the current image frame; stitch the motion estimation vector matrix of the target object and the static feature map of the current image frame based on the pixel points at the same position to obtain a primary stitching image; perform feature extraction on the primary stitching image based on a preset feature dimension to obtain a primary feature map of the primary stitching image; stitch the motion estimation vector matrix of the target object and the primary feature map based on the pixel points at the same position to obtain a secondary stitching image.
[0090] Optionally, the stitching module 906 is further used to: use a preset flame segmentation algorithm to perform feature extraction on the primary stitching image based on a preset feature dimension to obtain the primary feature image; wherein the preset flame segmentation algorithm includes at least one of the Deeplabv3 algorithm and the Unet algorithm.
[0091] Optionally, the detection module 908 is further used to: perform classification prediction on each pixel point in the secondary splicing image to determine a target area in the secondary splicing image that contains the target object; and identify a connected domain in the target area to obtain a detection result of the target object.
[0092] Optionally, the detection module 908 is further used to: perform morphological processing on the target area using a preset morphological processing algorithm; wherein the preset morphological processing algorithm includes at least one of an opening operation and a closing operation.
[0093] Optionally, the detection module 908 is also used to: perform edge detection on the connected domain based on a preset edge detection algorithm to determine the edge contour of the connected domain; perform rectangular fitting based on the edge contour of the connected domain to obtain the detection result of the target object; wherein the preset edge detection algorithm includes at least one of: a Canny algorithm and a Sobel algorithm.
[0094] Another embodiment of the present application further provides a computer storage medium, in which instructions for executing the steps in the above method embodiments are stored.
[0095] In summary, the target object detection method, device and computer storage medium provided in each embodiment of the present application can accurately detect the flame in the video to be detected, reduce the misjudgment rate of flame detection, and have little impact on the computational complexity of the model, thereby ensuring the detection efficiency of the model.
[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A target object detection method, comprising: Acquire multiple target image frames in the video to be detected; Determine a current image frame among the target image frames, detect multiple target image frames including the current image frame, and obtain dynamic features of the target object; splicing the dynamic features of the target object and the static features extracted from the current image frame to obtain splicing features; Performing classification prediction on the splicing features to obtain a target object detection result of the current image frame; Wherein, the step of splicing the dynamic features of the target object and the static features extracted from the current image frame to obtain the spliced features includes: According to the image channel of the current image frame, performing feature extraction on the current image frame to obtain a static feature map of the current image frame; Based on each pixel point at the same position, splicing the motion estimation vector matrix of the target object and the static feature map of the current image frame to obtain a spliced image; Based on a preset feature dimension, performing feature extraction on the primary spliced image to obtain a primary feature map of the primary spliced image; Based on each pixel point at the same position, the motion estimation vector matrix of the target object and the primary feature map are spliced to obtain a secondary spliced map.
2. The method according to claim 1, wherein: The step of acquiring a plurality of target image frames in the video to be detected includes: Based on a preset frame interval, sequentially extracting each original image frame in the video to be detected; A normalization process is performed on the pixel value of each pixel point in each original image frame to obtain a target image frame of each original image frame.
3. The method according to claim 2, wherein: After the step of sequentially extracting each original image frame in the video to be detected, the method further includes: The image size of each original image frame is adjusted based on a preset image size.
4. The method according to any one of claims 1 to 3, wherein: The target object includes flames, and the determining of a current image frame among the target image frames, detecting a plurality of target image frames including the current image frame, and obtaining dynamic features of the target object include: Determine a target image frame as a current image frame; Based on the current image frame, extracting a plurality of preceding image frames of the current image frame from each target image frame; Detecting the current image frame and each previous image frame to obtain a motion estimation vector matrix of the target object; Wherein, each preceding image frame is each target image frame whose time stamp is smaller than the time stamp of the current image frame, and each target image frame constituting each preceding image frame and the current image frame are sequentially continuous; The number of the extracted previous image frames is between 4 and 19; Each matrix element in the motion estimation vector matrix is used to identify the motion estimation value of each pixel in the current image frame.
5. The method according to claim 4, wherein: The detecting the current image frame and each previous image frame to obtain a motion estimation vector matrix of the target object includes: Detecting the current image frame and each previous image frame using an optical flow method or a frame difference method to obtain a motion estimation vector matrix of the target object; Wherein, each matrix element in the motion estimation vector matrix is used to identify the grayscale instantaneous change rate value or grayscale difference value of each pixel point in the current image frame.
6. The method according to claim 1, wherein: The step of performing feature extraction on the primary spliced image based on a preset feature dimension to obtain a primary feature image of the primary spliced image includes: Using a preset flame segmentation algorithm to perform feature extraction on the primary splicing image based on a preset feature dimension to obtain the primary feature image; The preset flame segmentation algorithm includes at least one of the Deeplabv3 algorithm and the Unet algorithm.
7. The method according to claim 1, wherein: The performing classification prediction on the splicing features to obtain the target object detection result of the current image frame includes: Performing classification prediction for each pixel in the secondary spliced image to determine a target area in the secondary spliced image that contains the target object; A connected domain in the target area is identified to obtain a detection result of the target object.
8. The method according to claim 7, wherein: The method further comprises: Performing morphological processing on the target area using a preset morphological processing algorithm; Wherein, the preset morphological processing algorithm includes at least one of an opening operation and a closing operation.
9. The method according to claim 7 or 8, wherein: The identifying a connected domain in the target area to obtain a detection result of the target object includes: Performing edge detection on the connected domain based on a preset edge detection algorithm to determine the edge contour of the connected domain; Performing rectangle fitting based on the edge contour of the connected domain to obtain a detection result of the target object; The preset edge detection algorithm includes at least one of a Canny algorithm and a Sobel algorithm.
10. A target object detection device, comprising: An acquisition module is used to acquire multiple target image frames in the video to be detected; A feature analysis module, used to determine a current image frame among the target image frames, detect multiple target image frames including the current image frame, and obtain dynamic features of the target object; A splicing module, used for splicing the dynamic features of the target object and the static features extracted from the current image frame to obtain a splicing feature; A detection module, used to perform classification prediction on the splicing features to obtain a detection result of the target object in the current image frame; Wherein, the step of splicing the dynamic features of the target object and the static features extracted from the current image frame to obtain the spliced features includes: According to the image channel of the current image frame, performing feature extraction on the current image frame to obtain a static feature map of the current image frame; Based on each pixel point at the same position, splicing the motion estimation vector matrix of the target object and the static feature map of the current image frame to obtain a spliced image; Based on a preset feature dimension, performing feature extraction on the primary spliced image to obtain a primary feature map of the primary spliced image; Based on each pixel point at the same position, the motion estimation vector matrix of the target object and the primary feature map are spliced to obtain a secondary spliced map.
11. A computer storage medium, characterized in that: The computer storage medium stores instructions for executing the steps in the method according to any one of claims 1 to 9.