Target object detection method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides a target object detection method, device, storage medium and electronic device, the method comprising: obtaining a group of continuous frame images, extracting the x-th frame to be detected from the group of continuous frame images, and extracting the first n frames of the x-th frame to be detected from the group of continuous frame images according to a predetermined timing information dependency length n, where n is an integer greater than 1 and less than x, inputting the x-th frame to be detected and the first n frames into a target neural network model to obtain a target output result. Therefore, the technical problems that the target detection process in the related technology is relatively complicated and the utilization efficiency is low in actual application scenarios can be solved, so as to achieve the technical effect of improving the detection efficiency of the target object and reducing the detection cost.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a target object detection method, device, storage medium and electronic device. Background Art

[0002] With the rapid expansion of urbanization and economic growth, the number of urban roads, highways, tunnels and vehicles has increased dramatically, which has increased the workload of processing video streams generated by cameras on traffic roads, and has exceeded the manual load. To solve this problem, various object detection algorithms for video streams have emerged and have been applied in many practical scenarios. The current mainstream video object detection algorithms can be mainly divided into two categories, namely, video object detection algorithms that integrate optical flow and deep learning, and video object detection algorithms that integrate object tracking and static image detection.

[0003] For example, the video target detection algorithm based on the fusion of optical flow and deep learning lacks full utilization of contextual information in the video stream during application, and cannot effectively avoid missed and false detections of targets caused by changes in target posture, lighting, etc. in adjacent frames. At the same time, the time consumption introduced by optical flow calculation will limit the popularization and application of the algorithm.

[0004] For example, the video target detection algorithm based on the fusion of target tracking and static image detection contains a relatively complex algorithm structure during the application process, which also limits its further application in real scenes.

[0005] There is no effective solution to the problem that the target detection process in related technologies is relatively complicated and has low utilization efficiency in actual application scenarios. Summary of the invention

[0006] The embodiments of the present invention provide a target object detection method, device, storage medium and electronic device to at least solve the technical problems existing in the related art that the target detection process is relatively complicated and has low utilization efficiency in actual application scenarios.

[0007] According to one embodiment of the present invention, a target object detection method is provided, comprising: acquiring a group of continuous frame images; extracting an x-th frame of image to be detected from the group of continuous frame images, and extracting the first n frames of image of the x-th frame of image to be detected from the group of continuous frame images according to a predetermined timing information dependency length n, wherein n is an integer greater than 1 and less than x; inputting the x-th frame of image to be detected and the first n frames of image into a target neural network model to obtain a target output result, wherein the target output result is marked with the target border, and the target neural network model is used to determine the target border from the x-th frame of image to be detected based on the semantic information contained in the first n frames of image, and the target border corresponds to the target object contained in the x-th frame of image.

[0008] According to another embodiment of the present invention, a target object detection device is provided, including: an acquisition module, used to acquire a group of continuous frame images; an extraction module, used to extract the x-th frame of the picture to be detected from the group of continuous frame images, and extract the first n frames of the picture of the x-th frame to be detected from the group of continuous frame images according to a predetermined timing information dependency length n, wherein n is an integer greater than 1 and less than x; a processing module, used to input the x-th frame of the picture to be detected and the first n frames of the picture into a target neural network model to obtain a target output result, wherein the target output result is marked with the target border, and the target neural network model is used to determine the target border from the x-th frame of the picture to be detected based on the semantic information contained in the first n frames of the picture, and the target border corresponds to the target object contained in the x-th frame of the picture.

[0009] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps in any of the above method embodiments when executed by a processor.

[0010] According to another embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in any one of the above method embodiments when executing the computer program.

[0011] Through the present invention, a group of continuous frame images are obtained, the x-th frame to be detected is extracted from the group of continuous frame images, and the first n frames of the x-th frame to be detected are extracted from the group of continuous frame images according to a predetermined timing information dependency length n, where n is an integer greater than 1 and less than x, the x-th frame to be detected and the first n frames are input into a target neural network model to obtain a target output result, so as to determine a target bounding box corresponding to a target object contained in the x-th frame image, and to detect the target object. Therefore, the technical problems that the target detection process in the related art is relatively complicated and the utilization efficiency is low in actual application scenarios can be solved, and the technical effect of improving the detection efficiency of the target object and reducing the detection cost can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0013] Figure 1 is a hardware structure block diagram of a mobile terminal according to an optional target object detection method of an embodiment of the present invention;

[0014] Figure 2 is a flow chart of an optional target object detection method according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of an optional target object detection method according to an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of another optional target object detection method according to an embodiment of the present invention;

[0017] Figure 5 is a flow chart of another optional target object detection method according to an embodiment of the present invention;

[0018] Figure 6 is a structural block diagram of an optional target object detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0021] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a target object detection method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0022] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the target object detection method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0023] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0024] In this embodiment, a target object detection method running on a mobile terminal, a computer terminal or a similar computing device is provided. Figure 2FIG. 1 is a flow chart of an optional target object detection method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0025] S202, obtaining a set of continuous frame images;

[0026] S204, extracting the x-th frame to be detected from a group of continuous frame images, and extracting the first n frames of the x-th frame to be detected from a group of connected frame images according to a predetermined time sequence information dependency length n, where n is an integer greater than 1 and less than x;

[0027] S206, input the x-th frame of the picture to be detected and the previous n frames of the picture into the target neural network model to obtain a target output result, wherein the target output result is marked with a target bounding box, and the target neural network model is used to determine the target bounding box from the x-th frame of the picture to be detected based on the semantic information contained in the previous n frames of the picture, and the target bounding box corresponds to the target object contained in the x-th frame of the picture.

[0028] Optionally, in this embodiment, the above-mentioned set of continuous frame images may include but are not limited to a set of continuous frame images obtained after photographing the target area, or include but are not limited to a set of continuous frame images obtained from a database. The above-mentioned target area may include but are not limited to roads, highways, tunnels, bridges and other areas where there are target objects that need to be detected. The above-mentioned target objects may include but are not limited to people, cars, animals, robots, ships, aircraft and other objects that need to be identified and detected.

[0029] Optionally, in this embodiment, the above x is used to represent the frame number in the above set of continuous frame pictures, and the above x-th frame picture may include but is not limited to the current frame being the x-th frame to be detected, or the x-th frame picture preset by the system or server.

[0030] Optionally, in this embodiment, the above-mentioned group of continuous frame pictures may include but is not limited to a group of pictures obtained by intercepting or extracting frames from a video.

[0031] The acquisition of a set of continuous frame images can be achieved by including but not limited to the following methods:

[0032] S1, analyzing the current scene and extracting continuous frames of the collected video data to obtain the above-mentioned set of continuous frame images;

[0033] S2: label the above set of continuous frame images according to actual needs.

[0034] Optionally, in this embodiment, the extracting of the x-th frame to be detected from the group of continuous frame pictures, and extracting the first n frames of the x-th frame to be detected from the group of connected frame pictures according to the predetermined timing information dependency length n can be implemented including but not limited to the following methods:

[0035] S1, obtaining a predetermined timing information dependency length n, wherein the timing information dependency length can be flexibly adjusted according to actual conditions.

[0036] S2, when it is necessary to detect the target object in a set of continuous frame images, extract the static image of the current frame and the static image of the previous n frames according to the actual temporal information dependency length n, divide them into two paths, and set them as the size of the network input, where the size of the first path is B×C×H×W, and the size of the second path is B×C×D×H×W, B represents the number of input feature maps, C represents the number of input feature map channels, D represents the temporal dependency length n, H represents the height of the input feature map, and W represents the width of the input feature map.

[0037] It should be noted that the value range of n can be configured according to actual needs, and is generally an integer greater than 1 and less than the frame number x of the current frame image to be detected.

[0038] Optionally, in this embodiment, the above-mentioned target neural network model may include but is not limited to a convolutional neural network, an unsupervised pre-trained network, etc. Specifically, it may include but is not limited to, for example, a lightweight backbone network of yolov3, etc.

[0039] Optionally, in this embodiment, when the above-mentioned target neural network model is a convolutional neural network model, it may include but is not limited to a two-dimensional convolutional neural network model or a three-dimensional convolutional neural network model, and a combination of a two-dimensional convolutional neural network model and a three-dimensional convolutional neural network model.

[0040] Optionally, in this embodiment, the semantic information contained in the first n frames of pictures may include but is not limited to inputting the first n frames of pictures into a three-dimensional convolutional neural network model to determine the semantic information contained in the first n frames of pictures, and the above semantic information may include but is not limited to temporal semantic information, etc.

[0041] Optionally, in this embodiment, the target frame is used to identify the target object in the corresponding image to detect the target object.

[0042] Through the present invention, a group of continuous frame images are obtained, the x-th frame to be detected is extracted from the group of continuous frame images, and the first n frames of the x-th frame to be detected are extracted from the group of continuous frame images according to a predetermined timing information dependency length n, where n is an integer greater than 1 and less than x, the x-th frame to be detected and the first n frames are input into a target neural network model to obtain a target output result, so as to determine a target bounding box corresponding to a target object contained in the x-th frame image, and to detect the target object. Therefore, the technical problems that the target detection process in the related art is relatively complicated and the utilization efficiency is low in actual application scenarios can be solved, and the technical effect of improving the detection efficiency of the target object and reducing the detection cost can be achieved.

[0043] In an optional embodiment, the x-th frame of the picture to be detected and the first n frames of the picture are input into the target neural network model to obtain a target output result, including: inputting the x-th frame of the picture to be detected into a two-dimensional convolutional neural network model to obtain a first output result, wherein the target neural network model includes the two-dimensional convolutional neural network model; inputting the first n frames of the picture into a three-dimensional convolutional neural network model to obtain a second output result, wherein the target neural network model includes the three-dimensional convolutional neural network model; inputting the first output result and the second output result into the objective function to obtain the target output result.

[0044] Optionally, in this embodiment, the above-mentioned two-dimensional convolutional neural network model may include but is not limited to a two-dimensional convolutional neural network model determined by a lightweight backbone network based on yolov3, and the above-mentioned three-dimensional convolutional neural network model may include but is not limited to a three-dimensional convolutional neural network model determined by a lightweight backbone network based on yolov3.

[0045] Optionally, in this embodiment, the above-mentioned first output result may include but is not limited to the output result after directly performing the convolution and pooling operation, and may also include but is not limited to the output result obtained by inputting the output result after the convolution and pooling operation into the feature pyramid module.

[0046] Optionally, in this embodiment, the above-mentioned second output result may include but is not limited to the output result after directly performing the convolution and pooling operation, and may also include but is not limited to the output result after performing the convolution and pooling operation and then upsampling and convolution operation.

[0047] The above is only an example, and this embodiment does not make any specific limitation.

[0048] In an optional embodiment, the x-th frame of the image to be detected is input into a two-dimensional convolutional neural network model to obtain a first output result, including: when the two-dimensional convolutional neural network model includes M convolutional pooling modules and a separate convolutional module, the x-th frame of the image to be detected is input into the first convolutional pooling module among the M convolutional pooling modules to obtain the output result of the M-th convolutional pooling module among the M convolutional pooling modules, wherein, among the M convolutional pooling modules, the output of the previous convolutional pooling module is the input of the next convolutional pooling module; the output result of the M-th convolutional pooling module is input into the separate convolutional module to obtain a third output result; the third output result and the output result of the M-2-th convolutional pooling module among the M convolutional pooling modules are input into a feature pyramid module to obtain a fourth output result; and the third output result and the fourth output result are determined as the first output result.

[0049] Optionally, in this embodiment, the above M can be flexibly set according to actual needs.

[0050] For example, when the above-mentioned two-dimensional convolutional neural network model adopts the lightweight backbone network of yolov3 and M is equal to 6, the above-mentioned two-dimensional convolutional neural network model may include but is not limited to being composed of 6 convolutional pooling modules and a single convolution module, wherein the convolution kernel size of the convolution module in the first 5 convolutional pooling modules is 3×3, the step size is 1, the maximum pooling module kernel size is 2×2, the step size is 2, the last convolutional pooling module has a convolution kernel size of 3×3, the step size is 1, the maximum pooling module kernel size is 2×2, the step size is 2, and the single convolution module has a convolution kernel size of 3×3, and the step size is 1.

[0051] In other words, the above-mentioned two-dimensional convolutional neural network model may include but is not limited to M-2 first-type convolutional pooling modules, one second-type convolutional pooling module and a separate convolution module, and the convolution kernel size and step size of the above-mentioned first-type convolutional pooling module and the second-type convolutional pooling module are different.

[0052] The above is only an example, and this embodiment does not make any specific limitation to this.

[0053] Optionally, in this embodiment, taking M=6 as an example, the third output result and the output result of the fourth convolutional pooling module among the six convolutional pooling modules are input into the feature pyramid module to obtain the first output result.

[0054] In an optional embodiment, the first n frames of images are input into a three-dimensional convolutional neural network model to obtain a second output result, including: when the three-dimensional convolutional neural network model includes N convolutional pooling modules and a separate convolutional module, the first n frames of images are respectively input into the first convolutional pooling module among the N convolutional pooling modules to obtain the output result of the Nth convolutional pooling module among the N convolutional pooling modules, wherein, among the N convolutional pooling modules, the output of the previous convolutional pooling module is the input of the next convolutional pooling module; the output result of the Nth convolutional pooling module is input into the separate convolutional module to obtain a fifth output result; upsampling and convolution operations are performed on the fifth output result to obtain a sixth output result; and the fifth output result and the sixth output result are determined as the second output result.

[0055] Optionally, in this embodiment, the above N can be flexibly set according to actual needs.

[0056] For example, in the case where N of the above-mentioned three-dimensional convolutional neural network model is equal to 5, it may include but is not limited to being composed of 5 convolutional pooling modules and a single convolution module, wherein the convolution kernel size of the first two convolutional pooling modules is 3×3×3, the step size is 1, the convolution kernel size of the pooling module is 3×3×3, and the step size is 2, the convolution kernel size of the last three convolutional pooling modules is 3×3, the step size is 1, the convolution kernel size of the pooling module is 3×3, the step size is 2, and the convolution kernel size of the single convolution module is 3×3, and the step size is 1.

[0057] In other words, the above-mentioned three-dimensional convolutional neural network model may include but is not limited to a group of third-type convolutional pooling modules and a group of fourth-type convolutional pooling modules and a single convolution module, and the convolution kernels and strides of the above-mentioned third-type convolutional pooling modules and fourth-type convolutional pooling modules are different.

[0058] Optionally, in this embodiment, the above M and N can be preset to be the same or different.

[0059] Through this embodiment, the use of a three-dimensional convolution kernel can fully utilize the spatiotemporal features in a set of continuous frame images to achieve the technical effect of improving the detection efficiency of the target object.

[0060] In an optional embodiment, inputting the first output result and the second output result into an objective function to obtain the target output result includes: inputting the first output result and the second output result into the following objective function to obtain the target output result:

[0061]

[0062] Wherein, x1 represents the second output result, x2 represents the first output result, and both x1 and x2 satisfy x∈R c×h×w , c represents the number of channels of the input feature map, h represents the height of the input feature map, and w represents the width of the input feature map. represents the Hadamard product, and the objective function includes an identity mapping module.

[0063] Optionally, in this embodiment, "1" in the above objective function represents an identity mapping module included in the above objective function, so as to achieve the preservation of the semantic features of the above x-th frame, ie, the current frame to be detected, as much as possible.

[0064] In an optional embodiment, after inputting the x-th frame of the image to be detected and the previous n frames of the image into the target neural network model to obtain the target output result, the method also includes: performing recognition and border regression convolution on the target output result to obtain a first border; performing candidate target post-processing on the first border to determine the target border and the object category corresponding to the target border.

[0065] Optionally, in this embodiment, the size information of the first bounding box is determined by inputting pre-annotated bounding box information into an anchor box obtained by a clustering algorithm.

[0066] In an optional embodiment, the method further includes: annotating the group of continuous frame images according to a predetermined annotation strategy to obtain a group of continuous frame images annotated with predetermined borders; inputting the predetermined borders into a target clustering algorithm to obtain an anchor frame corresponding to the predetermined border, wherein the anchor frame is used to determine the size information of the first border before performing candidate target post-processing on the first border.

[0067] Optionally, in this embodiment, the target clustering algorithm may include but is not limited to a K-means clustering algorithm.

[0068] Optionally, in this embodiment, taking the above-mentioned two-dimensional convolutional neural network model as an example, the lightweight backbone network of yolov3 is used, and the target border is clustered using the K-means clustering algorithm. The number of cluster centers may include but is not limited to being set to 6, and the corresponding evaluation function is:

[0069] d(gt_box,c_box)=1-IOU(gt_box,c_box) (2)

[0070] Among them, gt_box and c_box represent the annotated bounding box and cluster center bounding box respectively, and the function IOU represents the intersection-over-union ratio. When performing bounding box regression, each point on the feature map is regarded as a detection target center, and a horizontal and vertical offset is predicted for each center point. For the size of the anchor point, the corresponding width and height scaling ratio is predicted, that is, t∈R 4 , as shown in formula (3):

[0071] b x =σ(t x )+c x

[0072] b y =σ(t y )+c y

[0073]

[0074]

[0075] After obtaining the downsampling multiples corresponding to each feature map, a relatively complete border corresponding to the original image can be obtained. When predicting the corresponding category for each center point, since each input feature map contains multiple targets, the Sigmoid function is used for prediction, and the cross entropy loss function is used for back propagation during training.

[0076] In an optional embodiment, after determining the target bounding box and the object category, the method further includes: using non-maximum suppression to remove redundant bounding boxes contained in the target bounding box to obtain a target detection result, wherein the target detection result is used to represent the target object contained in the x-th frame image.

[0077] Optionally, in this embodiment, after obtaining the category and bounding box of the target object, redundant frames may be removed by, including but not limited to, non-maximum suppression (NMS), to obtain the above target detection result.

[0078] Figure 3 is a schematic diagram of an optional target object detection method according to an embodiment of the present invention, such as Figure 3 As shown, the airplane 302 is the aforementioned target object, and the frame 304 is the aforementioned target frame.

[0079] Figure 4 is a schematic diagram of another optional target object detection method according to an embodiment of the present invention. Figure 4As shown, the target objects and target frames shown in sub-image 402, sub-image 404, sub-image 406, sub-image 408, sub-image 410, sub-image 412, sub-image 414, sub-image 416, sub-image 418, and sub-image 420 are all target detection results obtained after the sub-image is used as the aforementioned x-th frame image for target object detection.

[0080] The above is only an example, and this embodiment does not make any specific limitation.

[0081] The present invention is further explained below with reference to specific examples:

[0082] S1. Based on the analysis of complex video stream traffic scenes, the collected video data is extracted into continuous frames, and the data set is labeled according to actual needs.

[0083] S2. When detecting the target of the current frame of the video stream, extract the static image of the current frame and the static images of the previous n frames according to the actual temporal information dependency length n, divide them into two paths, and set them to the network input size; the size of the first path is B×C×H×W, and the size of the second path is B×C×D×H×W; B represents the number of input feature maps, C represents the number of input feature map channels, D represents the temporal dependency length n, H represents the height of the input feature map, and W represents the width of the input feature map.

[0084] S3. Send the image input obtained in step 2 to the 2DCNN network and the 3DCNN network respectively. The 2DCNN network adopts the lightweight backbone network of yolov3, which is composed of 6 convolutional pooling modules and a single convolutional module. The convolution kernel size of the first 5 convolutional pooling modules is 3×3, the step size is 1, the maximum pooling module kernel size is 2×2, and the step size is 2. The convolution kernel size of the last convolutional pooling module is 3×3, the step size is 1, the maximum pooling module kernel size is 2×2, and the step size is 2. The length is 2; 3DCNN uses 3D convolution kernels to make full use of the spatiotemporal features in the video stream. Its specific structure consists of 5 convolutional pooling modules and a single convolutional module. The convolution kernel size of the first two convolutional pooling modules is 3×3×3, the step size is 1, and the convolution kernel size of the pooling module is 3×3×3, the step size is 2. The convolution kernel size of the last three convolutional pooling modules is 3×3, the step size is 1, the convolution kernel size of the pooling module is 3×3, the step size is 2, and the convolution kernel size of the single convolution module is 3×3, the step size is 1. The final output feature map of 2DCNN and the convolution module output feature map in the fourth convolutional pooling module are sent to the FPN (Feature Pyramid Networks for Object Detection) module to obtain two feature outputs respectively as the first and second outputs of the network; the final output of 3DCNN is used as the first output of the network, and its first output is used as the second output of the network after 2 times upsampling and convolution operation.

[0085] S4. After the first and second outputs of 3DCNN pass through the designed spatiotemporal attention model, the final input of the detector is obtained. There is a high similarity between adjacent frames in the video stream, and it also contains semantic information that plays a vital role in target detection. Therefore, the first few frames of the frame to be detected are used to predict and enhance the semantic information of the current frame. At the same time, in order to preserve the semantic features of the current frame as much as possible, an identity mapping module is added to the spatiotemporal attention model, so a point-level spatiotemporal attention model is designed. The corresponding specific formula can be expressed as follows:

[0086]

[0087] Wherein, x1 represents the second output result, x2 represents the first output result, and both x1 and x2 satisfy x∈R c×h×w , c represents the number of channels of the input feature map, h represents the height of the input feature map, and w represents the width of the input feature map. represents the Hadamard product, and the objective function includes an identity mapping module. Each feature generated by the spatiotemporal attention module is processed through recognition and bounding box regression convolution and candidate target post-processing to generate the final target bounding box.

[0088] S5. Since the anchor points listed in the original YOLOV3 algorithm cannot be effectively applied to the targets in this video traffic dataset, the K-means clustering algorithm is used to cluster the target borders of this dataset. The number of cluster centers is 6, and the corresponding evaluation function is:

[0089] d(gt_box,c_box)=1-IOU(gt_box,c_box) (2)

[0090] Among them, gt_box and c_box represent the annotated bounding box and cluster center bounding box respectively, and the function IOU represents the intersection-over-union ratio. When performing bounding box regression, each point on the feature map is regarded as a detection target center, and a horizontal and vertical offset is predicted for each center point. For the size of the anchor point, the corresponding width and height scaling ratio is predicted, that is, t∈R 4 , as shown in formula (3):

[0091] b x =σ(t x )+c x

[0092] b y =σ(t y )+c y

[0093]

[0094]

[0095] S6, after obtaining the downsampling multiple corresponding to each feature map, the corresponding relatively complete border in the original image can be obtained. When predicting the corresponding category for each center point, since each input feature map contains multiple targets, the Sigmoid function is used for prediction, and the cross entropy loss function is used for back propagation during training. After obtaining the category and border of the target, the non-maximum suppression NMS (Non Maximum Suppression) is used to remove redundant frames to obtain the final detection result.

[0096] Through this embodiment, a point-to-point spatiotemporal attention model is designed according to actual traffic application scenarios. It mainly predicts the temporal semantic information of the current frame by the previous frames of the current frame to be detected, and fully retains the language features of the current frame through the identity mapping module. The 3DCNN network topology is refined by analyzing the target attributes in the traffic scene, and the video stream feature extraction structure integrated with the 2DCNN network and FPN module; the overall process structure of the video stream target detection method designed according to the actual traffic application scenario can improve the robustness of the target object detection algorithm for target feature extraction in complex video stream traffic scenes, and can also effectively extract the deep features of the frame to be detected and its corresponding temporal features through a carefully designed lightweight 3DCNN feature extraction network and a lightweight 2DCNN feature extraction network; it can effectively improve the performance of target detection in video stream traffic scenes, alleviate the missed detection, false detection and false detection of single-frame target detection in the video, not only effectively utilize the language information of the previous frames of the current frame to be detected to predict and enhance the semantic information of the current frame, but also fully preserve the deep semantic features of the current frame by adding the identity mapping module to the spatiotemporal attention model method.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0098] In the present embodiment, a target object detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0099] Figure 6 is a structural block diagram of an optional target object detection device according to an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0100] An acquisition module 602 is used to acquire a set of continuous frame images;

[0101] An extraction module 604 is used to extract an x-th frame to be detected from the set of continuous frame images, and extract the first n frames of the x-th frame to be detected from the set of continuous frame images according to a predetermined time sequence information dependency length n, where n is an integer greater than 1 and less than x;

[0102] The processing module 606 is used to input the x-th frame of the picture to be detected and the first n frames of the picture into the target neural network model to obtain a target output result, wherein the target output result is marked with the target border, and the target neural network model is used to determine the target border from the x-th frame of the picture to be detected based on the semantic information contained in the first n frames of the picture, and the target border corresponds to the target object contained in the x-th frame of the picture.

[0103] In an optional embodiment, the processing module 606 is used to input the x-th frame of the image to be detected and the first n frames of the image into the target neural network model to obtain the target output result in the following manner:

[0104] Inputting the x-th frame of the image to be detected into a two-dimensional convolutional neural network model to obtain a first output result, wherein the target neural network model includes the two-dimensional convolutional neural network model;

[0105] Inputting the first n frames of images into a three-dimensional convolutional neural network model to obtain a second output result, wherein the target neural network model includes the three-dimensional convolutional neural network model;

[0106] The first output result and the second output result are input into an objective function to obtain the target output result.

[0107] In an optional embodiment, the processing module 606 is used to input the x-th frame of the image to be detected into a two-dimensional convolutional neural network model to obtain a first output result in the following manner:

[0108] In the case where the two-dimensional convolutional neural network model includes M convolutional pooling modules and a separate convolutional module, the x-th frame of the image to be detected is input into the first convolutional pooling module among the M convolutional pooling modules to obtain the output result of the M-th convolutional pooling module among the M convolutional pooling modules, wherein, among the M convolutional pooling modules, the output of the previous convolutional pooling module is the input of the next convolutional pooling module; the output result of the M-th convolutional pooling module is input into the separate convolutional module to obtain a third output result; the third output result and the output result of the M-2-th convolutional pooling module among the M convolutional pooling modules are input into the feature pyramid module to obtain a fourth output result; the third output result and the fourth output result are determined as the first output result.

[0109] In an optional embodiment, the processing module 606 is used to input the first n frames of images into the three-dimensional convolutional neural network model in the following manner to obtain a second output result: when the three-dimensional convolutional neural network model includes N convolutional pooling modules and a separate convolutional module, the first n frames of images are respectively input into the first convolutional pooling module among the N convolutional pooling modules to obtain the output result of the Nth convolutional pooling module among the N convolutional pooling modules, wherein, among the N convolutional pooling modules, the output of the previous convolutional pooling module is the input of the next convolutional pooling module; the output result of the Nth convolutional pooling module is input into the separate convolutional module to obtain a fifth output result; upsampling and convolution operations are performed on the fifth output result to obtain a sixth output result; the fifth output result and the sixth output result are determined as the second output result.

[0110] In an optional embodiment, the processing module 606 is used to input the first output result and the second output result into an objective function in the following manner to obtain the target output result:

[0111] The first output result and the second output result are input into the following objective function to obtain the target output result:

[0112]

[0113] Wherein, x1 represents the second output result, x2 represents the first output result, and both x1 and x2 satisfy x∈R c×h×w , c represents the number of channels of the input feature map, h represents the height of the input feature map, and w represents the width of the input feature map. represents the Hadamard product, and the objective function includes an identity mapping module.

[0114] In an optional embodiment, the device is also used to: after inputting the x-th frame of the image to be detected and the previous n frames of the image into the target neural network model to obtain the target output result, the target output result is processed using recognition and border regression convolution and candidate target post-processing to obtain the target border.

[0115] In an optional embodiment, the device is also used to: after obtaining the target border through recognition and border regression convolution and post-processing of candidate targets, the target output result is input into a target clustering algorithm to obtain the object category corresponding to the target border; after determining the target border and the object category, non-maximum suppression is used to remove redundant borders to obtain a target detection result, wherein the target detection result is used to represent the target object contained in the x-th frame image.

[0116] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0117] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0118] In this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0119] S1, obtain a set of continuous frame images;

[0120] S2, extracting the x-th frame to be detected from a group of continuous frame images, and extracting the first n frames of the x-th frame to be detected from a group of connected frame images according to a predetermined time sequence information dependency length n, where n is an integer greater than 1 and less than x;

[0121] S3, inputting the x-th frame of the image to be detected and the previous n frames of the image into the target neural network model to obtain a target output result, wherein the target output result is marked with a target bounding box, and the target neural network model is used to determine the target bounding box from the x-th frame of the image to be detected based on the semantic information contained in the previous n frames of the image, and the target bounding box corresponds to the target object contained in the x-th frame of the image.

[0122] The computer readable storage medium is further configured to store a computer program for performing the following steps:

[0123] S1, obtain a set of continuous frame images;

[0124] S2, extracting the x-th frame to be detected from a group of continuous frame images, and extracting the first n frames of the x-th frame to be detected from a group of connected frame images according to a predetermined time sequence information dependency length n, where n is an integer greater than 1 and less than x;

[0125] S3, inputting the x-th frame of the image to be detected and the previous n frames of the image into the target neural network model to obtain a target output result, wherein the target output result is marked with a target bounding box, and the target neural network model is used to determine the target bounding box from the x-th frame of the image to be detected based on the semantic information contained in the previous n frames of the image, and the target bounding box corresponds to the target object contained in the x-th frame of the image.

[0126] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0127] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0128] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0129] In an exemplary embodiment, the processor may be configured to perform the following steps via a computer program:

[0130] S1, obtain a set of continuous frame images;

[0131] S2, extracting the x-th frame to be detected from a group of continuous frame images, and extracting the first n frames of the x-th frame to be detected from a group of connected frame images according to a predetermined time sequence information dependency length n, where n is an integer greater than 1 and less than x;

[0132] S3, inputting the x-th frame of the image to be detected and the previous n frames of the image into the target neural network model to obtain a target output result, wherein the target output result is marked with a target bounding box, and the target neural network model is used to determine the target bounding box from the x-th frame of the image to be detected based on the semantic information contained in the previous n frames of the image, and the target bounding box corresponds to the target object contained in the x-th frame of the image.

[0133] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0134] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A target object detection method, characterized in that: include: Get a set of continuous frame images; Extracting an x-th frame to be detected from the set of continuous frame images, and extracting n frames before the x-th frame to be detected from the set of continuous frame images according to a predetermined timing information dependency length n, where n is an integer greater than 1 and less than x; Inputting the x-th frame of the image to be detected and the first n frames of the image into the target neural network model to obtain a target output result, wherein the target output result is marked with a target border, and the target neural network model is used to determine the target border from the x-th frame of the image to be detected based on the semantic information contained in the first n frames of the image, and the target border corresponds to the target object contained in the x-th frame of the image; The step of inputting the x-th frame of the image to be detected and the first n frames of the image into the target neural network model to obtain the target output result includes: inputting the x-th frame of the image to be detected into a two-dimensional convolutional neural network model to obtain a first output result, wherein the target neural network model includes the two-dimensional convolutional neural network model; inputting the first n frames of the image into a three-dimensional convolutional neural network model to obtain a second output result, wherein the target neural network model includes the three-dimensional convolutional neural network model; inputting the first output result and the second output result into the target function to obtain the target output result; the step of inputting the x-th frame of the image to be detected into the two-dimensional convolutional neural network model to obtain the first output result includes: inputting the first n frames of the image to be detected into the two-dimensional convolutional neural network model to obtain the first output result, wherein the two-dimensional convolutional neural network model includes M In the case of a convolution pooling module and a separate convolution module, the x-th frame of the image to be detected is input into the first convolution pooling module among the M convolution pooling modules to obtain the output result of the M-th convolution pooling module among the M convolution pooling modules, wherein, among the M convolution pooling modules, the output of the previous convolution pooling module is the input of the next convolution pooling module; the output result of the M-th convolution pooling module is input into the separate convolution module to obtain a third output result; the third output result and the output result of the M-2-th convolution pooling module among the M convolution pooling modules are input into the feature pyramid module to obtain a fourth output result; the third output result and the fourth output result are determined as the first output result.

2. The method according to claim 1, characterized in that Inputting the first n frames of images into a three-dimensional convolutional neural network model to obtain a second output result, including: In the case where the three-dimensional convolutional neural network model includes N convolutional pooling modules and a separate convolutional module, the first n frames of images are respectively input into the first convolutional pooling module of the N convolutional pooling modules to obtain the output result of the Nth convolutional pooling module of the N convolutional pooling modules, wherein, in the N convolutional pooling modules, the output of the previous convolutional pooling module is the input of the next convolutional pooling module; Inputting the output result of the Nth convolutional pooling module into the single convolutional module to obtain a fifth output result; Performing upsampling and convolution operations on the fifth output result to obtain a sixth output result; The fifth output result and the sixth output result are determined as the second output result.

3. The method according to claim 1, characterized in that Inputting the first output result and the second output result into an objective function to obtain the target output result includes: The first output result and the second output result are input into the following objective function to obtain the target output result: Wherein, x1 represents the second output result, x2 represents the first output result, and both x1 and x2 satisfy x∈R c×h×w , c represents the number of channels of the input feature map, h represents the height of the input feature map, and w represents the width of the input feature map. represents the Hadamard product, and the objective function includes an identity mapping module.

4. The method according to claim 1, characterized in that: After inputting the x-th frame of the image to be detected and the first n frames of the image into the target neural network model to obtain the target output result, the method further includes: Perform recognition and border regression convolution on the target output result to obtain a first border; The first frame is subjected to candidate target post-processing to determine the target frame and the object category corresponding to the target frame.

5. The method according to claim 4, characterized in that The method further comprises: Annotating the set of continuous frame images according to a predetermined annotation strategy to obtain a set of continuous frame images annotated with a predetermined border; The predetermined bounding box is input into a target clustering algorithm to obtain an anchor box corresponding to the predetermined bounding box, wherein the anchor box is used to determine size information of the first bounding box before performing candidate target post-processing on the first bounding box.

6. The method according to claim 4, characterized in that After determining the target bounding box and the object category, the method further includes: Non-maximum suppression is used to remove redundant borders contained in the target border to obtain a target detection result, wherein the target detection result is used to represent the target object contained in the x-th frame image.

7. A target object detection device, characterized in that: include: An acquisition module is used to acquire a set of continuous frame images; An extraction module, configured to extract an x-th frame of the image to be detected from the set of continuous frame images, and extract n frames of images before the x-th frame of the image to be detected from the set of continuous frame images according to a predetermined time sequence information dependency length n, where n is an integer greater than 1 and less than x; A processing module, used for inputting the x-th frame of the picture to be detected and the first n frames of the picture into a target neural network model to obtain a target output result, wherein the target output result is marked with a target border, and the target neural network model is used for determining the target border from the x-th frame of the picture to be detected based on the semantic information contained in the first n frames of the picture, and the target border corresponds to the target object contained in the x-th frame of the picture; The device is used to input the x-th frame of the picture to be detected and the first n frames of the picture into the target neural network model to obtain the target output result in the following manner: input the x-th frame of the picture to be detected into the two-dimensional convolutional neural network model to obtain a first output result, wherein the target neural network model includes the two-dimensional convolutional neural network model; input the first n frames of the picture into the three-dimensional convolutional neural network model to obtain a second output result, wherein the target neural network model includes the three-dimensional convolutional neural network model; input the first output result and the second output result into the target function to obtain the target output result; the inputting the x-th frame of the picture to be detected into the two-dimensional convolutional neural network model to obtain the first output result includes: in the two-dimensional convolutional neural network model In the case of including M convolutional pooling modules and a separate convolutional module, the x-th frame of the image to be detected is input into the first convolutional pooling module among the M convolutional pooling modules to obtain the output result of the M-th convolutional pooling module among the M convolutional pooling modules, wherein, among the M convolutional pooling modules, the output of the previous convolutional pooling module is the input of the next convolutional pooling module; the output result of the M-th convolutional pooling module is input into the separate convolutional module to obtain a third output result; the third output result and the output result of the M-2-th convolutional pooling module among the M convolutional pooling modules are input into the feature pyramid module to obtain a fourth output result; the third output result and the fourth output result are determined as the first output result.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 6 when executed by a processor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 6 are implemented.

Images