A target detection method, device, equipment and medium

Abstract

The embodiments of the present application disclose a target detection method, device, equipment and medium, and relate to the field of computer vision technology. The method includes: in the training stage of the detection network, information entropy calculation is performed on the feature vectors in the specified area near the annotation box of the training image to determine the information entropy threshold near the annotation box. The information entropy threshold characterizes the environmental features around the annotation box, so that after the detection network converges, the feature vectors in the target area of ​​the first image are identified through the video to be processed, and then the average information entropy in the target area is determined by calculating the information entropy of the feature vectors. The above average information entropy characterizes the environmental features around the first detection object. According to the comparison result of the average information entropy and the information entropy threshold, it can be determined whether the environmental features around the first detection object are similar to the environmental features around the annotation box, and then it is determined whether the first detection object is the target to be searched.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and specifically to a target detection method, device, equipment and medium. Background Art

[0002] Urban appearance management is an important indicator of modern smart city construction, and illegal placement of billboards is a major problem in urban appearance management. Currently, most control personnel conduct patrol inspections on video images captured by city camera points to manually monitor whether there are illegally placed billboards near the points. The above method requires high labor costs and has low processing efficiency.

[0003] With the development of science and technology, the detection task of illegal billboards has been developed in the industry. In related technologies, the detection model is often trained so that the model has the ability to automatically determine whether there are illegally placed billboards in the input image after convergence. However, the current detection models for illegal billboards often have the following problems: due to the complex street environment where the billboards are located, the common algorithm model has poor robustness in complex street scenes and cannot make accurate judgments on whether the billboards are illegal. Algorithm models with higher accuracy are usually too large and complex, and are not suitable for deployment in embedded devices, and have the problem of low applicability. Summary of the invention

[0004] The embodiments of the present application provide a target detection method, apparatus, device and medium for determining whether a billboard is illegally placed according to the average information entropy of feature vectors near the billboard in the image after determining the location of the billboard in the image through a detection model.

[0005] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising:

[0006] Inputting the video to be detected into the trained detection network, identifying each video image in the video to be detected through the detection network, and obtaining a first image containing a first prediction frame; the first image is a video image containing the first detection object in the video to be detected, and the first prediction frame represents a position area of ​​the first detection object in the first image;

[0007] For each first image, determine the average information entropy of the feature vector according to the feature vector of the target area in the first image; wherein the feature vector is obtained by extracting features of the first image through the detection network; and the target area is determined according to the first prediction box;

[0008] Based on the comparison result of the average information entropy and the information entropy threshold, determine whether the first detection object in the first image is the target to be searched; wherein, the information entropy threshold is obtained by calculating the information entropy of the feature vectors in the specified area of ​​the training image during the training process of the detection network, and the specified area is determined according to the annotation box in the training image; the target to be searched is the first detection object of the specified category.

[0009] In the embodiment of the present application, during the training phase of the detection network, information entropy calculation is performed on the feature vectors in the specified area near the annotation box of the training image to determine the information entropy threshold near the annotation box. The information entropy threshold characterizes the environmental features around the annotation box, so that after the detection network converges, the feature vectors in the target area of ​​the first image are identified through the video to be processed, and then the average information entropy in the target area is determined by calculating the information entropy of the feature vectors. The above average information entropy characterizes the environmental features around the first detection object, and the comparison result between the average information entropy and the information entropy threshold can determine whether the environmental features around the first detection object are similar to the environmental features around the annotation box, and then determine whether the first detection object is the target to be searched.

[0010] In some possible embodiments, a specified number of target areas are set around the first prediction box, and the size of each target area is determined based on the size of the first prediction box;

[0011] The determining the average information entropy according to the feature vector of the target area in the first image includes:

[0012] Obtain the feature vector of each target area respectively;

[0013] For each target region, the average information entropy of the target region is determined according to the feature vector in the target region.

[0014] In some possible embodiments, determining whether the first detection object in the first image is a target to be searched based on a comparison result of the average information entropy and an information entropy threshold includes:

[0015] If there are a specified number of target areas whose average information entropy is greater than the information entropy threshold corresponding to the target area, it is determined that the first detection object is not the target to be searched.

[0016] In some possible embodiments, the detection network is further used to identify whether the input image contains a second detection object, and determine a second prediction box representing a position of the second detection object in the input image; the input image includes the video image;

[0017] After determining whether the first detection object in the first image is a target to be searched based on the comparison result of the average information entropy and the information entropy threshold, the method further includes:

[0018] Determine an intersection-and-union ratio between the first prediction box and the second prediction box in the second image; wherein the second image is a first image in which the first detection object is determined to be the target to be searched based on the comparison result;

[0019] Based on the intersection-over-union ratio, it is determined whether the first detected object in the second image is the target to be searched, and the second image in which the first detected object is the target to be searched is output and displayed.

[0020] In some possible embodiments, determining whether the first detection object in the second image is the target to be searched based on the intersection-over-union ratio includes:

[0021] Determine a comparison image according to the time sequence position of the video image corresponding to the second image in the video to be detected;

[0022] Determine whether the first detection object is the target to be searched according to the intersection-and-union ratio between the first prediction box and the second prediction box in the comparison image.

[0023] In some possible embodiments, determining the comparison image according to the time sequence position of the second image corresponding to the video image in the video to be detected includes:

[0024] Determine a preset frame image whose timing is before the video image and / or whose timing is after the video image;

[0025] The first image corresponding to the preset frame image is used as the comparison image.

[0026] In a second aspect, an embodiment of the present application provides a target detection device, the device comprising:

[0027] The image recognition module is configured to input the video to be detected into the trained detection network, and recognize each video image in the video to be detected through the detection network to obtain a first image containing a first prediction frame; the first image is a video image containing the first detection object in the video to be detected, and the first prediction frame represents a position area of ​​the first detection object in the first image;

[0028] an information entropy acquisition module, configured to determine, for each first image, an average information entropy of the feature vector according to a feature vector of a target area in the first image; wherein the feature vector is obtained by extracting features from the first image through the detection network; and the target area is determined according to the first prediction box;

[0029] The target discrimination module is configured to determine whether the first detection object in the first image is a target to be searched based on the comparison result of the average information entropy and the information entropy threshold; wherein the information entropy threshold is obtained by calculating the information entropy of the feature vectors in the specified area of ​​the training image during the training process of the detection network, and the specified area is determined according to the annotation box in the training image; the target to be searched is the first detection object of the specified category.

[0030] In some possible embodiments, a specified number of target areas are set around the first prediction box, and the size of each target area is determined based on the size of the first prediction box;

[0031] To perform the step of determining the average information entropy according to the feature vector of the target area in the first image, the information entropy acquisition module is configured as follows:

[0032] Obtain the feature vector of each target area respectively;

[0033] For each target region, the average information entropy of the target region is determined according to the feature vector in the target region.

[0034] In some possible embodiments, the step of determining whether the first detection object in the first image is a target to be searched based on the comparison result between the average information entropy and the information entropy threshold is performed, and the target discrimination module is configured as follows:

[0035] If there are a specified number of target areas whose average information entropy is greater than the information entropy threshold corresponding to the target area, it is determined that the first detection object is not the target to be searched.

[0036] In some possible embodiments, the detection network is further used to identify whether the input image contains a second detection object, and determine a second prediction box representing a position of the second detection object in the input image; the input image includes the video image;

[0037] After executing the determination of whether the first detection object in the first image is a target to be searched based on the comparison result of the average information entropy and the information entropy threshold, the target discrimination module is further configured to:

[0038] Determine an intersection-and-union ratio between the first prediction box and the second prediction box in the second image; wherein the second image is a first image in which the first detection object is determined to be the target to be searched based on the comparison result;

[0039] Based on the intersection-over-union ratio, it is determined whether the first detected object in the second image is the target to be searched, and the second image in which the first detected object is the target to be searched is output and displayed.

[0040] In some possible embodiments, the determining whether the first detected object in the second image is the target to be searched based on the intersection-over-union ratio is performed, and the target discrimination module is configured as follows:

[0041] Determine a comparison image according to the time sequence position of the video image corresponding to the second image in the video to be detected;

[0042] Determine whether the first detection object is the target to be searched according to the intersection-and-union ratio between the first prediction box and the second prediction box in the comparison image.

[0043] In some possible embodiments, the determination of the comparison image according to the temporal position of the second image corresponding to the video image in the video to be detected is performed, and the target discrimination module is configured as follows:

[0044] Determine a preset frame image whose timing is before the video image and / or whose timing is after the video image;

[0045] The first image corresponding to the preset frame image is used as the comparison image.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0047] A memory for storing program instructions;

[0048] The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods in the first aspect according to the obtained program instructions.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods described in the first aspect.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product comprises: a computer program code, and when the computer program code runs on a computer, the computer executes any one of the methods described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a designated area provided for an embodiment of the present application;

[0052] Figure 2 A system architecture diagram provided for an embodiment of the present application;

[0053] Figure 3 An overall flow chart of a target detection algorithm provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a target area provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the information entropy threshold of a non-violation type billboard provided in an embodiment of the present application;

[0056] Figure 6 The intersection-and-union ratio diagram provided in the embodiment of the present application;

[0057] Figure 7 A schematic diagram of a video image including multiple prediction boxes provided in an embodiment of the present application;

[0058] Figure 8 A flow chart of dynamic inspection and filtering provided in an embodiment of the present application;

[0059] Fig. 9 A structural diagram of a target detection device 900 provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0062] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0063] As mentioned above, in the related art, the detection model is often trained so that after the model converges, it has the ability to automatically determine whether there are billboards placed illegally in the input image. The detection model is trained mainly by marking illegal billboards in the training image so that the detection model has the ability to identify illegal billboards in the input image.

[0064] However, in actual applications, billboards are usually set up in street environments with dense traffic and strong commerciality. There is a high probability that multiple illegal billboards and non-illegal billboards will appear in a video image. For example, in actual street monitoring scenes, common floor-standing billboards mainly include light boxes, vertical billboards, roll-up billboards, neon lights, and various types of display board billboards, etc. Most of these billboards are illegally placed. Billboards fixed on motor vehicles and non-motor vehicles, store lintels, store walls, electric poles / street lights, bus stations, etc. are non-illegal floor-standing billboards. Ordinary algorithm models have poor robustness in complex street scenes. When detecting illegal and non-illegal billboards on the street at the same time, they cannot distinguish between the two well. Algorithm models with higher accuracy are usually too large and complex, and are not suitable for deployment in embedded devices, and have the problem of low applicability.

[0065] In order to solve the above problems, the inventive concept of the present application is as follows: during the training phase of the detection network, information entropy calculation is performed on the feature vectors in a specified area near the annotation box of the training image to determine the information entropy threshold near the annotation box. The information entropy threshold characterizes the environmental features around the annotation box, so that after the detection network converges, the feature vectors in the target area of ​​the first image are identified through the video to be processed, and then the average information entropy in the target area is determined by calculating the information entropy of the feature vectors. The above average information entropy characterizes the environmental features around the first detection object, and the comparison result between the average information entropy and the information entropy threshold can determine whether the environmental features around the first detection object are similar to the environmental features around the annotation box, and then determine whether the first detection object is the target to be searched.

[0066] To facilitate understanding of the technical solution provided by the present application, the training process of the detection model of the present application is first briefly introduced. In the training stage of the detection network, the embodiment of the present application pre-marks the location of the billboard in the training image, and the marked box is the real box (gt) set for the training image. Then, by inputting the training image into the detection network to be trained, the network model is corrected according to the classification loss and coordinate regression loss between the prediction box (anchor) and the real box determined by the network model parameters, and the training image is recognized again using the corrected network model, and the above process is iterated until the classification loss and coordinate regression loss corresponding to the corrected network model meet the preset loss threshold, at which time the model converges.

[0067] The above training process is not used to train the detection model to have the ability to identify whether a billboard is illegal. It is only necessary to train the detection model to have the basic requirements of identifying whether there is a billboard in the input image and the location area of ​​the billboard in the image. Therefore, the above training process can be implemented using traditional target detection algorithms such as yolo, faster-rnn, etc., without any improvement to the algorithm, and will not increase additional requirements for algorithm deployment.

[0068] In the above-mentioned training stage, after the detection network performs convolution processing on the training image to obtain a feature map, the embodiment of the present application calculates the information entropy of the feature vectors extracted from the specified area near the real frame in the feature map. The information entropy can characterize the surrounding environment information of the illegal billboard in the real frame in the real scene. Figure 1 It shows the feature map obtained by convolution processing of the training image by the detection network during the training phase, such as Figure 1 As shown in the shaded part, the real frame of the feature map has corresponding designated areas in the four directions of top, bottom, left and right. As mentioned above, billboards fixed on motor vehicles and non-motor vehicles, store lintels, store walls, electric poles / street lights, bus stops, etc. are non-illegal ground billboards. By training a large number of training images for each type of non-illegal billboards mentioned above, the environmental features around each type of non-illegal billboards can be statistically analyzed. Thus, the information entropy threshold of the surrounding environmental features of each type of non-illegal billboards can be summarized. In this way, after the detection model converges, by determining the average information entropy of the surrounding environmental features of the billboard in the input image and comparing it with the information entropy threshold of each non-illegal billboard mentioned above, it can be determined whether the billboard in the input image is an illegal billboard based on the comparison result.

[0069] The following is a brief description of the deployment of the detection model for this application. Figure 2 As shown, the embodiment of the present application is based on deep learning target detection, and deploys a local space detection unit and a classification unit on the basis of the detection model, as well as a local information entropy filtering unit and a dynamic detection filtering unit for post-processing operations.

[0070] Among them, the local space detection unit is a unit for the detection model to learn the information around the billboard. Its main purpose is to extract the feature vector in the target area near the first prediction box of the feature map of the video image after the detection model performs convolution processing on the video image, that is, to obtain the local boundary feature information representing the surrounding environment of the billboard. Since the local boundary information contains rich location information, this can learn a set of adaptive thresholds (i.e., the above-mentioned information entropy thresholds) for the local information entropy filtering unit.

[0071] The classification unit mainly trains billboards separately to reduce the false detection of the detection model and further improve the accuracy of billboards. After the detection model outputs the prediction frame of the video image, the unit performs a secondary check on the prediction frame to determine whether the video image is the first image containing the billboard, and after determining the first image, informs the local information entropy filtering unit so that the local information entropy filtering unit calculates the average information entropy of the local boundary feature information of the first prediction frame, and then compares it with the above-mentioned information entropy threshold to determine whether the billboard in the first prediction frame is an illegal billboard.

[0072] In addition, considering that billboards carried by objects such as motor vehicles, non-motor vehicles and pedestrians are also non-illegal billboards, and since the carriers of such billboards are usually in motion, the environmental changes around the billboards vary greatly. Therefore, after the embodiment of the present application filters out non-illegal billboards with too low information entropy, such as those fixed on telephone poles / street lights, walls / store lintels, etc., through the above-mentioned local information entropy filtering unit, it is necessary to filter out the above-mentioned non-illegal billboards carried by mobile targets through the dynamic detection filtering unit.

[0073] Next, taking the above-mentioned application scenario of illegal billboard detection as an example, the process of each of the above-mentioned units executing a target detection method provided by this application is explained. Figure 3 As shown, including:

[0074] Step 301: inputting a video to be detected into a trained detection network, and using the detection network to identify each video image in the video to be detected to obtain a first image containing a first prediction frame; the first image is a video image in the video to be detected that contains the first detection object, and the first prediction frame represents a location area of ​​the first detection object in the first image;

[0075] The video to be detected is a video stream captured by a pre-fixed city camera within a specified time period. After the video to be detected is input into the trained detection network, the detection result of the detection network on the video image is obtained. The detection result is specifically characterized by adding a first prediction box output by the detection network to the video image. The target circled by the first prediction box is the billboard in the video image identified by the detection network, and the position area of ​​the first prediction box in the video image is the location of the billboard in the video image.

[0076] Thus, a first image including a billboard and a first prediction frame representing the area where the billboard is located in the first image can be selected from the video image.

[0077] Step 302: for each first image, determine the average information entropy of the feature vector according to the feature vector of the target area in the first image; wherein the feature vector is obtained by extracting features of the first image through the detection network; and the target area is determined according to the first prediction box;

[0078] After the video image is input into the detection model, the detection model performs convolution processing on the video image to obtain its corresponding feature map, and then determines whether the video image contains a billboard by performing feature recognition on the feature map.

[0079] After the detection model obtains the feature map of the video image, the local space detection unit obtains the local expansion information near the first prediction frame in the feature map, and the local expansion information represents the spatial characteristics of the surrounding environment of the billboard in the first prediction frame. Figure 4 As shown, according to the boundary position of the first prediction box, its local expansion can be divided into four target areas: upper, lower, left, and right (i.e., Figure 4 Assume that the local expansion information of the first prediction box captured in the video image is w g ,h g , the intercepted target is α and β times the length w and width h of the original target (α and β are both greater than 1), and the above w is obtained g =α×w,h g =β×h. Next, the feature maps w' and h' output after the network is downsampled by n times and the corresponding local expansion information size can be expressed as the following formula (1):

[0080]

[0081] From the above formula, it can be seen that the width and length of the upper, lower, left and right target areas are all (w g '-w') / 2 and (h g '-h') / 2. After the convolution feature map is downsampled by n times, it is reduced to a two-dimensional channel through a convolution kernel of a preset size. Through the above position transformation, the corresponding coordinates of the local expansion information of the upper, lower, left and right target areas on the feature maps of different downsampling ratios can be quickly obtained. Let function f be the information entropy function. According to the coordinate information of each target area, the average information entropy of the upper, lower, left and right four target areas on the two-dimensional channel can be calculated by the following formula (2):

[0082]

[0083] Where i, j is a tuple composed of pixel grayscales scaled to the range of 0 to 255, and its range is (i ≥ 0, j ≤ 255). The average information entropy of local pixels is used to characterize the spatial characteristics of the surrounding environment of the billboard.

[0084] Step 303: Determine whether the first detection object in the first image is the target to be searched based on the comparison result of the average information entropy and the information entropy threshold; wherein the information entropy threshold is obtained by calculating the information entropy of the feature vectors in the specified area of ​​the training image during the training process of the detection network, and the specified area is determined according to the annotation box in the training image; the target to be searched is the first detection object of the specified category.

[0085] The local entropy filtering unit calculates the information entropy of the target's surrounding environment to determine the specific location of the target and implement rule filtering. In the detection scenario of illegal billboards, billboards in different locations have similar characteristics. The filtering algorithm that introduces local entropy can make judgments by analyzing the nearby information of the billboards. In the embodiment of the present application, after obtaining the average information entropy of each target area through the above step 302, the local information entropy filtering unit determines whether the billboard in the first image is an illegal billboard based on the comparison result of the average information entropy and the information entropy threshold.

[0086] During implementation, first obtain the feature map and local expansion information, downsampling ratio and other parameters of the first image obtained through the detection network, and then post-process the features near the boundary of the first prediction box. Specifically, the information entropy of the feature vectors of the four target areas of the upper, lower, left and right of the first prediction box in the first image is calculated with the information entropy at the corresponding position of the feature map. In this way, the detection network can learn the feature information of the surrounding environment of the billboard while learning the features of the billboard in the above training stage. Specifically, take the average of the information entropy of the four target areas of the upper, lower, left and right of the non-illegal billboards of the same type in the training image, and obtain the corresponding information entropy as the information entropy threshold of the billboard of this category. The specific method of obtaining the information entropy threshold can be shown in the following formula (3).

[0087]

[0088] Among them, H is the information entropy of each target area. According to the above formula, the information entropy threshold parameters [σ, η, μ, λ] that characterize the four target areas of the upper, lower, left, and right of this category of billboards can be obtained, and then the parameters are updated by sliding average.

[0089] Specific as Figure 5 As shown, Figure 5The information entropy values ​​of each target area corresponding to different types of non-illegal billboards are shown, which reflects to a certain extent that the local information of different types of billboards is different. For example, assume that each type of non-illegal billboard is assigned a corresponding set of information entropy threshold parameters [σ, η, μ, λ]. After obtaining the average information entropy σ1, η1, μ1 and λ1 of each target area in the first image through the above step 302, the information entropy threshold parameters of each type are compared by comparing σ1 with σ, η1 with η, μ1 with μ, and λ1 with λ, and then determining whether the billboard in the first image is a non-illegal billboard of this type based on the comparison results.

[0090] During implementation, it can be determined that when the average information entropy of a specified number of target areas is greater than the information entropy threshold corresponding to the target area, it is determined that the first detection object is not the target to be searched. For example, if the average information entropy of at least three target areas in the first image satisfies Figure 5 If the information entropy threshold parameter of the "store lintel" type is met, the billboard in the first image is determined to be a non-illegal billboard of the "store lintel" type. Similarly, if the billboard in the first image does not meet the comparison of the information entropy threshold of each type of non-illegal billboard, the billboard in the first image can be temporarily determined to be an illegal billboard.

[0091] The reason for the provisional determination here is that, as mentioned above, billboards carried by motor vehicles, non-motor vehicles, pedestrians and other objects are also non-illegal billboards. Since the carriers of such billboards are usually in motion, the environmental changes around the billboards vary greatly. For this type of non-illegal billboards, the information entropy threshold comparison method cannot be used for effective identification. Therefore, it is inaccurate to determine whether a billboard is illegal only through the above-mentioned local information entropy filtering unit. At this time, the above-mentioned dynamic detection filtering unit is also required to filter out non-illegal billboards carried by mobile targets.

[0092] Specifically, the detection network of the embodiment of the present application can be trained in the same training method as the above-mentioned training billboard recognition capability during the training phase, so that it has the ability to recognize whether the input image contains a second detection object, and the second detection object is a common mobile target capable of carrying a billboard, such as a motor vehicle, a non-motor vehicle, and a pedestrian. In this way, after the above-mentioned video image is input into the trained detection network, the detection network can output a first prediction frame of the location of the billboard, and can also output a second prediction frame of the location of the second detection object such as a motor vehicle, a non-motor vehicle, and a pedestrian in the image.

[0093] In this way, after executing the above step 303 to determine whether the first detection object in the first image is the target to be searched based on the comparison result of the average information entropy and the information entropy threshold, the intersection-and-union ratio between the first prediction box and the second prediction box in the second image can be determined to determine whether the first detection object in the second image is the target to be searched based on the intersection-and-union ratio. The above second image is the first image in which the first detection object is determined to be the target to be searched based on the comparison result.

[0094] Here we first briefly introduce the intersection-over-union ratio (Iou), as follows Figure 6 As shown in , the intersection-over-union ratio is the ratio of the overlapping area between two anchor frames to the total area. Figure 6 A and B are two prediction boxes output by the detection network for different targets. The overlapping area between A and B is the black area shown in the figure, and the intersection-and-union ratio between the two is (A∩B) / (A∪B).

[0095] During implementation, the detection network will mark the detection results of the video image, that is, mark the first prediction box id of the billboard in the image and the second prediction box id of the moving target such as motor vehicle and non-motor vehicle, and then determine the comparison image according to the temporal position of the video image corresponding to the second image in the video to be detected, and then determine whether the first detection object is the target to be searched according to the intersection and union ratio between the first prediction box and the second prediction box in the comparison image.

[0096] Figure 7 The video image shown includes a first prediction frame including a billboard, a second prediction frame 1 including a motor vehicle, and a second prediction frame 2 including a non-motor vehicle. Figure 7 Taking the video image shown in FIG. 1 as an example, the judgment process performed by the above-mentioned motion detection filtering unit is described as follows: Figure 8 As shown, the following steps are included:

[0097] Step 801: Calculate the intersection-over-union ratio between the first prediction frame and each second prediction frame. Keep the ID pairs whose intersection-over-union ratio is greater than the threshold γ.

[0098] Assuming that the intersection-and-union ratio between the first prediction box and the second prediction box 2 is greater than the threshold γ, the IDs of the first prediction box and the second prediction box 2 are recorded to construct an ID pair.

[0099] Step 802: Track the video to be processed by t frames to determine the comparison image. Specifically, the video to be detected is tracked by t frames before and / or after, and the video image within t frames or the tth frame is used as the comparison image.

[0100] Step 803: Determine whether the position of the first prediction box in the comparison image has changed.

[0101] Step 804: If a change occurs, determine that the billboard in the first prediction frame is a non-illegal billboard.

[0102] Specifically, this indicates that the billboard is in a moving state. That is, the billboard in the first prediction frame is carried by the non-motor vehicle in the second prediction frame 2 and moves with the non-motor vehicle, and is a non-illegal billboard.

[0103] Step 805: If no change occurs, determine whether the intersection-over-union ratio of the ID pairs in the comparison image is greater than the threshold γ.

[0104] If there is no change, it means that the billboard has not moved. At this time, it is necessary to calculate whether the intersection and union ratio of the ID pair is still greater than the threshold γ for the stationary billboard. That is, calculate whether the intersection and union ratio between the first prediction box and the second prediction box 2 in the comparison image is still greater than the threshold γ.

[0105] Step 806: If it is not greater than the threshold γ, the billboard is determined to be an illegal billboard.

[0106] This indicates that the billboard in the first prediction frame is not carried by the non-motor vehicle in the second prediction frame 2, so it is an illegal billboard.

[0107] Step 807: If it is greater than the threshold γ, then based on the comparison image, the video to be processed is tracked for n times of t frames of the comparison image, wherein the above steps 803 to 805 are executed for each tracking.

[0108] This indicates that the billboard and the non-motor vehicle in the second prediction frame 2 may be in a relatively static state. To further improve the detection accuracy, the current comparison image can be used as a basis, and then the comparison image tracking of n t frames is performed to determine whether the intersection-over-union ratio between the first prediction frame and the second prediction frame 2 is greater than the threshold γ.

[0109] Step 808: Determine whether the intersection-over-union ratio within n tracking times is still greater than the threshold γ.

[0110] Step 809: If it is greater than the threshold γ, the billboard in the first prediction frame is determined to be a non-illegal billboard.

[0111] At this time, it is determined that the billboard and the non-motor vehicle in the first prediction frame are in a relatively stationary state, that is, the billboard is a non-illegal billboard.

[0112] Step 810: If it is not greater than the threshold γ, the billboard in the first prediction frame is determined to be an illegal billboard.

[0113] At this time, if any tracking within n tracking times finds that the intersection-and-union ratio between the billboard and the non-motor vehicle is not greater than the threshold γ, it can be said that the billboard is not carried by the non-motor vehicle, that is, the billboard is an illegal billboard.

[0114] It should also be noted that the local information entropy filtering unit in this article is used to determine whether the billboard is an illegal billboard based on the spatial feature information of the billboard's surrounding environment output by the detection model, and the motion detection filtering unit is used to detect whether there is a non-illegal billboard carried by a moving target in the video image. In the above description of the technical solution of this application, setting the motion detection filtering unit to perform a secondary judgment on the output result of the local information entropy filtering unit is only a detection method for whether the video to be processed contains illegal billboards.

[0115] In actual implementation, the motion detection filter unit may pre-filter the images of non-illegal billboards carried by the mobile target in the processed video, and then the local information entropy filter unit may perform a secondary judgment on the remaining images. The local information entropy filter unit and the motion detection filter unit may also perform parallel processing on each video image of the processed video, and then perform manual judgment based on the results of the two units, or set a judgment method such as weight, to filter out the final illegal billboards from the images containing illegal billboards output by the two units, and this application does not limit this.

[0116] Based on the same inventive concept, the present application embodiment provides a target detection device 900, specifically, Fig. 9 As shown, including:

[0117] The image recognition module 901 is configured to input the video to be detected into the trained detection network, and recognize each video image in the video to be detected through the detection network to obtain a first image containing a first prediction frame; the first image is a video image containing the first detection object in the video to be detected, and the first prediction frame represents a position area of ​​the first detection object in the first image;

[0118] The information entropy acquisition module 902 is configured to determine, for each first image, an average information entropy of the feature vector according to the feature vector of the target area in the first image; wherein the feature vector is obtained by extracting features from the first image by the detection network; and the target area is determined according to the first prediction box;

[0119] The target discrimination module 903 is configured to determine whether the first detection object in the first image is the target to be searched based on the comparison result of the average information entropy and the information entropy threshold; wherein the information entropy threshold is obtained by calculating the information entropy of the feature vectors in the specified area of ​​the training image during the training process of the detection network, and the specified area is determined according to the annotation box in the training image; the target to be searched is the first detection object of the specified category.

[0120] In some possible embodiments, a specified number of target areas are set around the first prediction box, and the size of each target area is determined based on the size of the first prediction box;

[0121] To perform the step of determining the average information entropy according to the feature vector of the target area in the first image, the information entropy acquisition module 902 is configured as follows:

[0122] Obtain the feature vector of each target area respectively;

[0123] For each target region, the average information entropy of the target region is determined according to the feature vector in the target region.

[0124] In some possible embodiments, the determination of whether the first detected object in the first image is a target to be searched based on the comparison result between the average information entropy and the information entropy threshold is performed, and the target discrimination module 903 is configured as follows:

[0125] If there are a specified number of target areas whose average information entropy is greater than the information entropy threshold corresponding to the target area, it is determined that the first detection object is not the target to be searched.

[0126] In some possible embodiments, the detection network is further used to identify whether the input image contains a second detection object, and determine a second prediction box representing a position of the second detection object in the input image; the input image includes the video image;

[0127] After executing the determination of whether the first detection object in the first image is a target to be searched based on the comparison result of the average information entropy and the information entropy threshold, the target discrimination module 903 is further configured to:

[0128] Determine an intersection-and-union ratio between the first prediction box and the second prediction box in the second image; wherein the second image is a first image in which the first detection object is determined to be the target to be searched based on the comparison result;

[0129] Based on the intersection-over-union ratio, it is determined whether the first detected object in the second image is the target to be searched, and the second image in which the first detected object is the target to be searched is output and displayed.

[0130] In some possible embodiments, the determining whether the first detected object in the second image is the target to be searched based on the intersection-over-union ratio is performed, and the target discrimination module 903 is configured as follows:

[0131] Determine a comparison image according to the time sequence position of the video image corresponding to the second image in the video to be detected;

[0132] Determine whether the first detection object is the target to be searched according to the intersection-and-union ratio between the first prediction box and the second prediction box in the comparison image.

[0133] In some possible embodiments, the target identification module 903 is configured to:

[0134] Determine a preset frame image whose timing is before the video image and / or whose timing is after the video image;

[0135] The first image corresponding to the preset frame image is used as the comparison image.

[0136] Refer to the following Fig.10 The electronic device 130 according to this embodiment of the present application is described. Fig.10 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0137] like Fig.10 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0138] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0139] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0140] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0141] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 132 including instructions, and the instructions can be executed by the processor 131 of the device 900 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0143] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, and when the computer program / instruction is executed by the processor 131, any one of the target detection methods provided in the present application is implemented.

[0144] In an exemplary embodiment, various aspects of a target detection method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a target detection method according to various exemplary embodiments of the present application described above in this specification.

[0145] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0146] The program product for performing target detection of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0147] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0148] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0149] The program code for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or completely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0150] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0151] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0153] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0155] These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0156] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0157] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A target detection method, characterized in that: The method comprises: Inputting the video to be detected into the trained detection network, identifying each video image in the video to be detected through the detection network, and obtaining a first image containing a first prediction frame; the first image is a video image containing a first detection object in the video to be detected, and the first prediction frame represents a position area of ​​the first detection object in the first image; For each first image, determine the average information entropy of the feature vector according to the feature vector of the target area in the first image; wherein the feature vector is obtained by extracting features of the first image through the detection network; and the target area is determined according to the first prediction box; Determine whether the first detection object in the first image is a target to be searched based on the comparison result of the average information entropy and the information entropy threshold; wherein the information entropy threshold is obtained by calculating the information entropy of the feature vectors in the specified area of ​​the training image during the training process of the detection network, and the specified area is determined according to the annotation box in the training image; the target to be searched is the first detection object of the specified category; Wherein, a specified number of target areas are arranged around the first prediction box, and the size of each target area is determined based on the size of the first prediction box; and determining the average information entropy according to the feature vector of the target area in the first image includes: The feature vectors of each target area are obtained respectively; for each target area, the average information entropy of the target area is determined according to the feature vectors in the target area.

2. The method according to claim 1, characterized in that The determining whether the first detection object in the first image is a target to be searched based on the comparison result of the average information entropy and the information entropy threshold comprises: If there are a specified number of target areas whose average information entropy is greater than the information entropy threshold corresponding to the target area, it is determined that the first detection object is not the target to be searched.

3. The method according to claim 1, characterized in that The detection network is further used to identify whether the input image contains a second detection object, and to determine a second prediction box representing a position of the second detection object in the input image; the input image includes the video image; After determining whether the first detection object in the first image is a target to be searched based on the comparison result of the average information entropy and the information entropy threshold, the method further includes: Determine an intersection-and-union ratio between the first prediction box and the second prediction box in the second image; wherein the second image is a first image in which the first detection object is determined to be the target to be searched based on the comparison result; Based on the intersection-over-union ratio, it is determined whether the first detected object in the second image is the target to be searched, and the second image in which the first detected object is the target to be searched is output and displayed.

4. The method according to claim 3, characterized in that The determining whether the first detection object in the second image is the target to be searched based on the intersection-over-union ratio includes: Determine a comparison image according to the time sequence position of the video image corresponding to the second image in the video to be detected; Determine whether the first detection object is the target to be searched according to the intersection-and-union ratio between the first prediction box and the second prediction box in the comparison image.

5. The method according to claim 4, characterized in that The step of determining a comparison image according to a time sequence position of a video image corresponding to the second image in the video to be detected includes: Determine a preset frame image whose timing is before the video image and / or whose timing is after the video image; The first image corresponding to the preset frame image is used as the comparison image.

6. A target detection device, characterized in that: The device comprises: The image recognition module is configured to input the video to be detected into the trained detection network, and recognize each video image in the video to be detected through the detection network to obtain a first image containing a first prediction frame; the first image is a video image containing a first detection object in the video to be detected, and the first prediction frame represents a position area of ​​the first detection object in the first image; an information entropy acquisition module, configured to determine, for each first image, an average information entropy of the feature vector according to a feature vector of a target area in the first image; wherein the feature vector is obtained by extracting features from the first image through the detection network; and the target area is determined according to the first prediction box; A target discrimination module is configured to determine whether the first detection object in the first image is a target to be searched based on a comparison result between the average information entropy and an information entropy threshold; wherein the information entropy threshold is obtained by calculating the information entropy of feature vectors in a specified area of ​​a training image during a training process of the detection network, and the specified area is determined based on a marked box in the training image; the target to be searched is a first detection object of a specified category; Wherein, a specified number of target areas are arranged around the first prediction box, and the size of each target area is determined based on the size of the first prediction box; and determining the average information entropy according to the feature vector of the target area in the first image includes: The feature vectors of each target region are obtained respectively; and for each target region, the average information entropy of the target region is determined according to the feature vectors in the target region.

7. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods of claims 1-5 according to the obtained program instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images