Target object detection method and device in image, storage medium and equipment

Abstract

The application discloses a target object detection method and device in an image, a storage medium and equipment, and belongs to the technical field of image processing. The method comprises the following steps: acquiring an image to be identified and a trained target detection model, wherein the target detection model at least comprises a feature extraction network and a Transformer module; performing feature extraction on the image by using the feature extraction network to obtain a feature map; generating a heat map according to the feature map, wherein the heat map comprises all potential target objects in the image; initializing each query point according to the heat map, wherein the query point represents the position of a potential target object in the image; and performing detection on the feature map and the initialized query points by using the Transformer module to obtain the target object in the image. The application can place a query point at the position of a potential target object shown in a heat map, thereby accelerating the inference speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a target object detection method and device in an image, a storage medium and equipment. BACKGROUND

[0002] A target object detection method based on deep learning refers to a method of identifying the category, position and size of a target object of interest in a given image based on a deep learning algorithm. Figure 1 As shown in the figure, two dogs, a person, a bag and a car in the image can be identified by the target object detection method.

[0003] A region convolutional neural network (R-CNN) in the deep learning algorithm adopts a two-stage method to realize a target object detection task, a first stage uses a heuristic method to find a region in which a target object may exist in an image, and a second stage uses a CNN to classify all potential regions to determine whether a target object actually exists.

[0004] However, the inference speed of the R-CNN is slow, resulting in slow detection speed of the target object. SUMMARY

[0005] The present application provides a target object detection method and device in an image, a storage medium and equipment, which are used to solve the problem of slow inference speed of the R-CNN, resulting in slow detection speed of the target object. The technical solution is as follows:

[0006] According to a first aspect of the present application, a target object detection method in an image is provided, and the method comprises the following steps:

[0007] obtaining an image to be identified and a trained target detection model, wherein the target detection model comprises at least a feature extraction network and a Transformer module;

[0008] extracting features of the image by using the feature extraction network to obtain a feature map;

[0009] generating a heat map according to the feature map, wherein the heat map contains all potential target objects in the image;

[0010] initializing each query point according to the heat map, wherein the query point represents the position of a potential target object in the image;

[0011] detecting the feature map and the initialized query points by using the Transformer module to obtain a target object in the image.

[0012] In a possible implementation, the initializing each query point according to the heat map comprises:

[0013] obtaining a preset assignment threshold;

[0014] assigning a position in the heat map with a value greater than the assignment threshold to each query point, and spacing different query points by n positions, n≥2.

[0015] In a possible implementation, the assigning the position in the heat map with the value greater than the assignment threshold to each query point comprises:

[0016] when traversing the value of each position in the heat map, if the value of the ith row and jth column is greater than the assignment threshold, assigning the position of the ith row and jth column to a query point, i and j are positive integers;

[0017] continuing to traverse the value in the heat map from a new position spaced n positions away from the position of the ith row and jth column.

[0018] In a possible implementation, the assigning the position in the heat map with the value greater than the assignment threshold to each query point further comprises:

[0019] if the value of the ith row and jth column is less than or equal to the assignment threshold, continuing to traverse the value in the heat map from a new position of the ith row and j+1th column.

[0020] In a possible implementation, the detecting the target object in the image by using the Transformer module on the feature map and the initialized query points comprises:

[0021] processing the feature map and the initialized query points by using multiple Transformer layers in the Transformer module, so that each query point gradually approaches the real position of the target object represented by the query point after being stacked by multiple Transformer layers, until each target object is detected.

[0022] According to a second aspect of the present application, an apparatus for detecting a target object in an image is provided, the apparatus comprising:

[0023] an obtaining module configured to obtain an image to be recognized and a trained target detection model, the target detection model comprising at least a feature extraction network and a Transformer module;

[0024] an extracting module configured to extract features of the image by using the feature extraction network to obtain a feature map;

[0025] generate a heat map according to the feature map, the heat map containing all potential target objects in the image;

[0026] initialize each query point according to the heat map, the query point representing a potential target object in the image;

[0027] detect the feature map and the initialized query points by using the Transformer module to obtain the target object in the image.

[0028] In a possible implementation, the initialization module is further configured to:

[0029] obtain a preset assignment threshold;

[0030] assign a position in the heat map with a value greater than the assignment threshold to each query point, and the different query points are spaced by n positions, n≥2.

[0031] In a possible implementation, the initialization module is further configured to:

[0032] when traversing the values of the positions in the heat map, if the value of the ith row and jth column is greater than the assignment threshold, the position of the ith row and jth column is assigned to a query point, i and j are positive integers;

[0033] continue to traverse the values in the heat map from a new position spaced by n positions from the position of the ith row and jth column.

[0034] According to a third aspect of the present application, a computer readable storage medium is provided, the storage medium stores at least one instruction, the at least one instruction is loaded and executed by a processor to implement the target object detection method in an image as described above.

[0035] According to a fourth aspect of the present application, a computer device is provided, the computer device comprises the target object detection device in an image.

[0036] The technical scheme provided by the present application has at least the following beneficial effects:

[0037] By generating a heat map of the potential target object according to the feature map of the image, and then initializing the positions of each query point (Query) based on the heat map, by placing the query point at the position of the potential target object, rather than randomly placing the query point as in the classic Deform-DETR, this approach not only reduces the number of invalid query points, but also filters out query points in regions without target objects, and each initialized query point is closer to the real target object, which can achieve the same effect with fewer Transformer layers and fewer query points, ultimately accelerating the inference speed of the model without sacrificing model performance, converging faster and consuming less computing power than the classic Deform-DETR. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0039] Figure 1 is a schematic diagram of the recognition result of a target object in an image;

[0040] Figure 2 is a comparison diagram of the recognition result and the heat map of the image;

[0041] Figure 3 is a schematic diagram of the Query gradually approaching the target object after multiple Transformer layer superposition processing;

[0042] Figure 4 is a flowchart of a target object detection method in an image provided by an embodiment of the present application;

[0043] Figure 5 is a flowchart of a target object detection method in an image provided by an embodiment of the present application;

[0044] Figure 6 is a schematic diagram of the processing flow of a target detection model provided by an embodiment of the present application;

[0045] Figure 7 is a structural block diagram of a target object detection device in an image provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0047] To accelerate the inference speed of R-CNN, Yolo and SSD propose a one-stage object detection scheme. Specifically, Yolo and SSD place a lot of anchor boxes in the image space, and then classify each anchor box (negative samples represent that the anchor box is not associated with any target object, and positive samples represent that the anchor box is associated with some target object). For the anchor box predicted as a positive sample, the error between the anchor box and the most matched target box is also predicted. Although Yolo and SSD have made great improvements in performance indicators and inference speed, this anchor-based scheme is more dependent on the placement of anchor boxes, and there is still a lot of room for improvement in performance indicators and inference speed.

[0048] To reduce the strong dependence of model performance on anchor boxes, CenterNet proposes a new one-stage object detection scheme. Specifically, CenterNet reformulates the object detection task as identifying the center position of the target object and regressing the size of the box. To achieve this goal, CenterNet uses a heat map to encode the center position of the target object, and then takes the features of the center position and decodes to get the size of the target object. As shown in Figure 2 , the left side of the figure represents an image that needs to be detected, and the right side of the figure represents a heat map, where each local maximum value represents the center position of an object. Compared with previous schemes, CenterNet is easier to train and has a significantly faster inference speed than previous methods.

[0049] DETR (Data-Efficient Image Transformer) directly predicts the top-left and top-right positions of the target object with the powerful representation ability of Transformer, and uses bipartite graph matching to calculate the training loss to train the model. Based on the innovative improvement, DETR has refreshed the results on the COCO dataset again. Although the architecture is elegant and the indicators are outstanding, the training speed of DETR is very slow, and the training resources consumed are more than 10 times those of previous methods.

[0050] Deform-DETR proposes a deformation attention mechanism and solves the problem of slow training convergence of DETR. Specifically, Deform-DETR randomly initializes some queries (Query), and each Query represents a potential target object in the image. By stacking Transformer layers, each Query will continuously move closer to the target object it represents, and its schematic diagram is shown in Figure 3 . Figure 3In the figure, star 0 represents the position of the query output by the Transformer at layer 0, star 1 represents the position of the query output by the Transformer at layer 1, star 2 represents the position of the query output by the Transformer at layer 2, and unnumbered stars represent the actual position of the target object. The query position is randomly initialized at layer 0 and gradually approaches the actual position of the target object after passing through each Transformer layer.

[0051] While Deform-DETR significantly improves upon the original DETR, its performance is significantly affected by the initial position of each query. Specifically, if the initial position of a query is far from the true target object, more Transformer layers are required to successfully predict the target object. Furthermore, placing queries in image regions devoid of target objects is a waste of resources.

[0052] To further improve the performance of Deform-DETR, this application uses heat maps to predict potential target objects in images and places queries at the locations of these potential target objects, rather than randomly placing queries as in Deform-DETR. This approach not only reduces the number of invalid queries but also simplifies the training difficulty of the model, allowing the same effect to be achieved with fewer Transformer layers and fewer queries, ultimately accelerating the model's inference speed without sacrificing model performance.

[0053] like Figure 4 As shown, it shows a method flow chart of a method for detecting target objects in an image provided by an embodiment of the present application. The method for detecting target objects in an image can be applied to a computer device. The method for detecting target objects in an image can include:

[0054] Step 401: Obtain an image to be recognized and a trained object detection model. The object detection model includes at least a feature extraction network and a Transformer module.

[0055] The image to be identified may be an image captured by a perception module in the vehicle, or an image acquired from other devices. The source of the image is not limited in this embodiment.

[0056] The target detection model can be a model improved based on Deform-DETR, which at least includes a feature extraction network and a Transformer module. The feature extraction network can be a CNN or a Transformer or other network.

[0057] Step 402, feature extraction is performed on the image by using a feature extraction network to obtain a feature map.

[0058] Step 403, a heat map is generated according to the feature map, and the heat map contains all potential target objects in the image.

[0059] In this embodiment, the idea of CenterNet can be used to generate a heat map according to the feature map. The value of each position in the heat map is in the interval [0, 1]. If there is no target object at a certain position in the heat map, the value of the position is 0. If there is a potential target object at a certain position in the heat map, the value of the position is greater than 0 and less than or equal to 1.

[0060] Step 404, each query point is initialized according to the heat map, and the query point represents the position of a potential target object in the image.

[0061] The purpose of initializing the query point according to the heat map is to place the query point at the position with a potential target object, which not only reduces the number of invalid query points, but also makes each initialized query point closer to the real target object.

[0062] The initialization method of the query point is described in detail below.

[0063] Step 405, the feature map and each initialized query point are detected by using a Transformer module to obtain the target object in the image.

[0064] The Transformer module includes multiple Transformer layers, and the feature map and each initialized query point can be detected by the stacked Transformer layers to obtain the target object in the image.

[0065] In summary, the target object detection method in the image provided by the embodiments of the present application generates a heat map of potential target objects according to the feature map of the image, and then initializes the positions of each query point (Query) based on the heat map. Instead of randomly placing query points as in the classic Deform-DETR, this approach not only reduces the number of invalid query points, but also filters query points in regions without target objects. Each initialized query point is closer to the real target object, which can achieve the same effect with fewer Transformer layers and fewer query points. Ultimately, the model inference speed is accelerated without sacrificing model performance, and the convergence is faster and the computing power consumption is smaller than the classic Deform-DETR.

[0066] For example, Figure 5As shown, it shows a flowchart of an image target object detection method provided by an embodiment of the present application, which can be applied in a computer device. The image target object detection method can include:

[0067] In step 501, an image to be identified and a trained target detection model are obtained, and the target detection model at least includes a feature extraction network and a Transformer module.

[0068] The image to be identified can be an image collected by a perception module in a vehicle, or an image obtained from other devices, and the source of the image is not limited in the embodiment.

[0069] The target detection model can be a model improved based on Deform-DETR, which at least includes a feature extraction network and a Transformer module, and the feature extraction network can be CNN or Transformer or other network.

[0070] In step 502, the feature extraction network is used to perform feature extraction on the image to obtain a feature map.

[0071] In step 503, a heat map is generated according to the feature map, and the heat map contains all potential target objects in the image.

[0072] In the embodiment, the idea of CenterNet can be used to generate the heat map according to the feature map. The value of each position in the heat map is in the interval [0, 1], if there is no target object in a certain position in the heat map, the value of the position is 0, if there is a potential target object in a certain position in the heat map, the value of the position is greater than 0 and less than or equal to 1.

[0073] In step 504, a preset assignment threshold is obtained; positions in the heat map with values greater than the assignment threshold are assigned to each query point, and different query points are separated by n positions, n≥2, and the query point represents the position of a potential target object in the image.

[0074] The purpose of initializing the query point according to the heat map is to place the query point at the position with the potential target object, which not only reduces the number of invalid query points, but also makes each initialized query point closer to the real target object.

[0075] Specifically, assigning positions in the heat map with values greater than the assignment threshold to each query point can include:

[0076] (1) When traversing the values of each position in the heat map, if the value of the i-th row and the j-th column is greater than the assignment threshold, the position of the i-th row and the j-th column is assigned to a query point, i and j are positive integers;

[0077] (2) Continue traversing the values ​​in the heat map from the new position that is n positions away from the position in the i-th row and j-th column;

[0078] (3) If the value in the i-th row and j-th column is less than or equal to the allocation threshold, continue traversing the values ​​in the heat map from the new position in the i-th row and j+1-th column.

[0079] Simply put, if a position is assigned to a query point, the traversal will continue at a new position that is n (n ≥ 2) positions away from the position; if a position is not assigned to a query point, the traversal will continue at a new position that is 1 position away from the position.

[0080] Step 505: Use the Transformer module to detect the feature map and each initialized query point to obtain the target object in the image.

[0081] The Transformer module includes multiple Transformer layers, which can detect the feature map and each initialized query point through the superimposed Transformer layers to obtain the target object in the image.

[0082] Specifically, a Transformer module is used to detect the feature map and each initialized query point to obtain the target object in the image, including: using multiple Transformer layers in the Transformer module to process the feature map and each initialized query point, so that after the multiple Transformer layers are superimposed, each query point gradually approaches the actual position of the target object it represents until each target object is detected.

[0083] like Figure 6 As shown in the figure, after an image is input into the target detection model, the feature extraction network CNN is first used to extract its features to obtain a feature map; then a heat map is generated based on the feature map, and each query point is initialized using the heat map; finally, the Transformer module is used to process the feature map and each query point to obtain the target objects in the image.

[0084] In summary, the method for detecting target objects in an image provided by the embodiments of the present application generates a heat map of potential target objects according to a feature map of the image, and then initializes the positions of each query point (Query) based on the heat map. Instead of randomly placing query points as in the classic Deform-DETR, the query points are placed at the positions of the potential target objects. This approach reduces the number of invalid query points, filters out query points in regions without target objects, and initializes each query point to be closer to the real target object. Fewer Transformer layers and fewer query points can achieve the same effect, accelerating the inference speed of the model without sacrificing model performance. Compared with the classic Deform-DETR, the method converges much faster and consumes less computing power.

[0085] As shown in Figure 7 , which shows a structural block diagram of a device for detecting target objects in an image provided by an embodiment of the present application. The device for detecting target objects in an image can be applied in a computer device. The device for detecting target objects in an image can include:

[0086] The acquisition module 710 is configured to acquire an image to be recognized and a trained target detection model, and the target detection model includes at least a feature extraction network and a Transformer module.

[0087] The extraction module 720 is configured to perform feature extraction on the image by using the feature extraction network to obtain a feature map.

[0088] The generation module 730 is configured to generate a heat map according to the feature map, and the heat map contains all potential target objects in the image.

[0089] The initialization module 740 is configured to initialize each query point according to the heat map, and the query point represents a potential target object in the image.

[0090] The detection module 750 is configured to detect the feature map and the initialized query points by using the Transformer module to obtain target objects in the image.

[0091] In an optional embodiment, the initialization module 740 is further configured to:

[0092] acquire a preset assignment threshold;

[0093] assign positions in the heat map with values greater than the assignment threshold to each query point, and different query points are spaced apart by n positions, n≥2.

[0094] In an optional embodiment, the initialization module 740 is further configured to:

[0095] When traversing the values in the heat map at each position, if the value in the i-th row and the j-th column is greater than the assignment threshold, the position in the i-th row and the j-th column is assigned to a query point, i and j are positive integers;

[0096] The values in the heat map are continued to be traversed from a new position that is spaced n positions away from the position in the i-th row and the j-th column.

[0097] In an optional embodiment, the initialization module 740 is further configured to:

[0098] If the value in the i-th row and the j-th column is less than or equal to the assignment threshold, the values in the heat map are continued to be traversed from a new position in the i-th row and the j+1-th column.

[0099] In an optional embodiment, the detection module 750 is further configured to:

[0100] The feature map and the initialized query points are processed by using the plurality of Transformer layers in the Transformer module, so that after being stacked by the plurality of Transformer layers, each query point gradually approaches the real position of the target object represented by the query point, until each target object is detected.

[0101] In summary, the target object detection device in an image provided by the embodiments of the present application generates a heat map of potential target objects according to a feature map of an image, and then initializes the positions of query points (Query) based on the heat map. Instead of randomly placing query points as in the classic Deform-DETR, this approach reduces the number of invalid query points, filters query points in regions without target objects, and initializes each query point to be closer to the real target object. Fewer Transformer layers and fewer query points can achieve the same effect, ultimately accelerating the inference speed of the model without sacrificing model performance. Compared with the classic Deform-DETR, the convergence is faster and the computing power consumption is smaller.

[0102] An embodiment of the present application provides a computer readable storage medium, the storage medium stores at least one instruction, the at least one instruction is loaded and executed by a processor to implement the target object detection method in an image as described above.

[0103] An embodiment of the present application provides a computer device, the computer device includes the target object detection device in an image.

[0104] It should be noted that the above embodiment provides the image target object detection device, when detecting the target object in the image, only the above-mentioned each function module is divided to illustrate, and in actual application, the above-mentioned function distribution can be completed by different function modules according to the needs, that is, the internal structure of the image target object detection device is divided into different function modules to complete all or part of the functions described above. In addition, the image target object detection device and the image target object detection method provided by the above embodiment belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0105] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program to instruct related hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0106] The above description does not limit the embodiments of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A method for detecting a target object in an image, characterized in that: The method comprises: Obtain an image to be recognized and a trained object detection model. The object detection model includes at least a feature extraction network and a Transformer module. The object detection model is an improved model based on Deform-DETR. Deform-DETR randomly initializes some query points, each of which represents a potential target object in the image. By stacking Transformer layers, each query point will continuously approach the target object it represents. Extracting features from the image using the feature extraction network to obtain a feature map; Generate a heat map based on the feature map, wherein the heat map includes all potential target objects in the image; wherein, if a target object does not exist at a certain position in the heat map, the value of the position is 0; if a target object may exist at a certain position in the heat map, the value of the position is greater than 0 and less than or equal to 1; Obtaining a preset allocation threshold; when traversing the values ​​of each position in the heat map, if the value of the i-th row, j-th column is greater than the allocation threshold, assigning the position of the i-th row, j-th column to a query point, where i and j are positive integers; continuing to traverse the values ​​in the heat map from a new position that is n positions away from the position of the i-th row, j-th column; if the value of the i-th row, j-th column is less than or equal to the allocation threshold, continuing to traverse the values ​​in the heat map from a new position of the i-th row, j+1-th column, with n positions between different query points, where n ≥ 2, and the position of the query point represents the position of a potential target object in the image; The feature map and each initialized query point are processed using multiple Transformer layers in the Transformer module, so that after being superimposed by multiple Transformer layers, each query point gradually approaches the true position of the target object it represents until each target object is detected.

2. A device for detecting a target object in an image, characterized in that: The device comprises: An acquisition module is used to obtain the image to be recognized and a trained target detection model. The target detection model includes at least a feature extraction network and a Transformer module. The target detection model is an improved model based on Deform-DETR. Deform-DETR randomly initializes some query points, each of which represents a potential target object in the image. By superimposing Transformer layers, each query point will continuously approach the target object it represents; An extraction module, configured to extract features from the image using the feature extraction network to obtain a feature map; a generating module, configured to generate a heat map based on the feature map, wherein the heat map includes all potential target objects in the image; wherein if a target object does not exist at a certain position in the heat map, the value of the position is 0; and if a target object may exist at a certain position in the heat map, the value of the position is greater than 0 and less than or equal to 1; An initialization module is configured to obtain a preset allocation threshold; when traversing the values ​​of various positions in the heat map, if the value of the i-th row, j-th column is greater than the allocation threshold, then assigning the position of the i-th row, j-th column to a query point, where i and j are positive integers; continuing to traverse the values ​​in the heat map from a new position that is n positions away from the position of the i-th row, j-th column; if the value of the i-th row, j-th column is less than or equal to the allocation threshold, then continuing to traverse the values ​​in the heat map from a new position of the i-th row, j+1-th column, with n positions between different query points, where n≥2, and the position of the query point represents the position of a potential target object in the image; A detection module is used to process the feature map and each initialized query point using multiple Transformer layers in the Transformer module so that after the multiple Transformer layers are superimposed, each query point gradually approaches the true position of the target object it represents until each target object is detected.

3. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for detecting a target object in an image according to claim 1.

4. A computer device, characterized in that: The computer device comprises the target object detection device in an image according to claim 2.

Images