Image detection method, device, equipment and storage medium

Abstract

Embodiments of the present application provide an image detection method, device and equipment and a storage medium, the method comprising: acquiring an image to be detected, the image comprising a target object, determining a first detection box corresponding to the target object according to a first feature map corresponding to the image; performing deformable convolution processing on the first feature map according to position information of the first detection box to obtain a second feature map; determining a second detection box corresponding to the target object according to the second feature map, and outputting the second detection. In the above scheme, the feature map used for target object detection is optimized through deformable convolution, so that more accurate detection results can be obtained.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image detection method, apparatus, device, and storage medium. Background Technology

[0002] With the practical application of Optical Character Recognition (OCR) technology, text detection tasks are encountered in many scenarios. The purpose of text detection is to detect the bounding boxes (called text boxes) of text content in an image.

[0003] After the location of the text box is determined, the image region corresponding to the text box is used as the recognition object for text recognition processing to extract the text content contained within the text box. Therefore, the accuracy of the text box detection result directly affects the accuracy of the subsequent text recognition result. Summary of the Invention

[0004] This invention provides an image detection method, apparatus, device, and storage medium that can accurately detect the location of a target object in an image.

[0005] In a first aspect, embodiments of the present invention provide an image detection method, the method comprising:

[0006] Acquire an image to be detected, wherein the image contains the target object;

[0007] A first detection box corresponding to the target object is determined based on the first feature map corresponding to the image;

[0008] Based on the position information of the first detection box, deformable convolution processing is performed on the first feature map to obtain the second feature map;

[0009] The second detection box corresponding to the target object is determined based on the second feature map;

[0010] Output the second detection box.

[0011] In a second aspect, embodiments of the present invention provide an image detection apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire an image to be detected, wherein the image contains the target object;

[0013] The detection module is configured to determine a first detection box corresponding to the target object based on a first feature map corresponding to the image; and to perform deformable convolution processing on the first feature map based on the position information of the first detection box to obtain a second feature map, and to determine a second detection box corresponding to the target object based on the second feature map.

[0014] The output module is used to output the second detection frame.

[0015] Thirdly, embodiments of the present invention provide an image detection method, the method comprising:

[0016] Receive a request from a user equipment to call the image detection service interface, the request including an image to be detected, the image containing a target object;

[0017] The following steps are performed using the processing resources corresponding to the image detection service interface:

[0018] A first detection box corresponding to the target object is determined based on the first feature map corresponding to the image;

[0019] Based on the position information of the first detection box, deformable convolution processing is performed on the first feature map to obtain the second feature map;

[0020] The second detection box corresponding to the target object is determined based on the second feature map;

[0021] The position information of the second detection frame is output to the user equipment.

[0022] Fourthly, embodiments of the present invention provide an image detection method, the method comprising:

[0023] Get product images;

[0024] The first text box is determined from the product image based on the first feature map corresponding to the product image;

[0025] The first feature map is subjected to deformable convolution processing based on the position information of the first text box to obtain the second feature map;

[0026] The second text box is determined from the product image based on the second feature map;

[0027] The text content extracted from the second text box is detected and processed.

[0028] Fifthly, embodiments of the present invention provide an electronic device, including: a memory and a processor; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the image detection method as described in the first or fourth aspect.

[0029] In a sixth aspect, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image detection method as described in the first or fourth aspect.

[0030] In the image detection scheme provided in this embodiment of the invention, during the target object detection process of an image, firstly, feature extraction is performed on the image to obtain a first feature map. Initial target detection processing is then performed based on the first feature map, i.e., determining the first detection box corresponding to each feature point in the first feature map. Next, the first feature map is optimized using deformable convolution, i.e., optimizing the feature values ​​of each feature point in the first feature map to obtain a second feature map. Specifically, taking the first detection box corresponding to a certain feature point as an example, deformable convolution is performed on the first feature map using the vertex coordinates of the first detection box to obtain new feature values ​​for the corresponding feature points in the second feature map, thus optimizing the feature values ​​of each feature point in the first feature map. Then, target detection processing is performed again based on the second feature map, i.e., determining the second detection box corresponding to each feature point in the second feature map. The position of the second detection box is used as the position of the target object in the image to be detected, and the second detection box is output, thus optimizing the initial detection result.

[0031] In the above scheme, the feature map used for target object detection is optimized by deformable convolution, thereby obtaining more accurate detection results. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart of an image detection method provided in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram illustrating a deformable convolution execution process provided in an embodiment of the present invention;

[0035] Figure 3a This is a schematic diagram of a text box detection result provided in an embodiment of the present invention;

[0036] Figure 3b This is a schematic diagram illustrating the output effect of an optimized detection box according to an embodiment of the present invention;

[0037] Figure 3c This is a schematic diagram illustrating the output effect of another optimized detection box provided in an embodiment of the present invention;

[0038] Figure 4 A flowchart of another image detection method provided in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram illustrating the meaning of the location-aware weight values ​​provided in an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of a frame merging process provided in an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram illustrating the application of an image detection method provided in an embodiment of the present invention;

[0042] Figure 8 A flowchart of another image detection method provided in an embodiment of the present invention;

[0043] Figure 9 A flowchart of another image detection method provided in an embodiment of the present invention;

[0044] Figure 10 A flowchart of another image detection method provided in an embodiment of the present invention;

[0045] Figure 11 This is a schematic diagram of the structure of an image detection device provided in an embodiment of the present invention;

[0046] Figure 12 To and Figure 11 The illustrated embodiment provides a schematic diagram of the electronic device corresponding to the image detection apparatus. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0049] The image detection method provided in this embodiment of the invention can be executed by an electronic device, which can be a terminal device such as a PC, laptop, or smartphone, or a server. The server can be a physical server containing an independent host, a virtual server, a cloud server, or a server cluster.

[0050] The main objective of the image detection method provided in this invention is to detect target objects in the image to be detected, that is, to detect the location of target objects contained in the image. Specifically, the location of the target object in the image to be detected can be represented by the position of the detection box surrounding the target object. In practical applications, the target objects to be detected can be text, people, vehicles, etc.

[0051] A traditional object detection method is based on bounding box regression, which suffers from inaccurate detection results. For example, in text detection scenarios, the boundaries of the detection boxes (i.e., text boxes in text detection scenarios) become even more inaccurate when encountering long text. To achieve accurate object detection, the solution provided in this embodiment of the invention is proposed.

[0052] The execution process of the image detection method provided herein is illustrated below with reference to the following embodiments.

[0053] Figure 1 A flowchart of an image detection method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes the following steps:

[0054] 101. Obtain the image to be detected, which contains the target object.

[0055] 102. Determine the first detection box corresponding to the target object based on the first feature map corresponding to the image.

[0056] 103. Based on the position information of the first detection box, perform deformable convolution processing on the first feature map to obtain the second feature map.

[0057] 104. Determine the second detection box corresponding to the target object based on the second feature map, and output the second detection box.

[0058] The image detection scheme provided in this invention follows this approach: First, features are extracted from the image to be detected to obtain an initial feature map (referred to as the first feature map). Based on the first feature map, an initial detection result, i.e., a first detection box, is obtained. Then, based on the initial text box detection result, deformable convolution is used to sample feature points on the first feature map, forming a new feature map (referred to as the second feature map) based on the sampled points, thus optimizing the feature map. Finally, the target object is detected based on the second feature map, ultimately obtaining a more accurate and optimized detection result, i.e., a second detection box.

[0059] It should be noted that a feature map contains several feature points. Assuming the spatial resolution of a feature map is H*W, this means that the feature map contains H*W feature points. Each feature point has two attributes: coordinates and eigenvalues. If the feature map is represented as a matrix of H*W elements, then the coordinates of a feature point can be considered to represent its row and column position in the matrix, and the eigenvalue of this feature point is the value of the element at that position in the matrix. The range of eigenvalues ​​is generally [0, 1]. In this embodiment of the invention, optimizing the first feature map refers to updating the eigenvalues ​​of its feature points. That is, the difference between the first and second feature maps is mainly that the eigenvalues ​​corresponding to feature points with the same coordinates will differ between the two feature maps.

[0060] The implementation process of the image detection scheme provided in the embodiments of the present invention will be described below.

[0061] When detecting target objects in an image, the first step is to extract features from the image to obtain a first feature map. This feature extraction can be achieved using network models such as convolutional neural networks. Then, target object detection is performed based on the first feature map to obtain initial detection results.

[0062] The process of object detection based on the first feature map can be simply described as follows: identify the category corresponding to each feature point in the first feature map—whether it corresponds to the target object; and for the feature points corresponding to the target object, determine the detection box corresponding to the feature point through bounding box regression.

[0063] In practical applications, the first feature map will contain multiple feature points that correspond to the target object. Each feature point will regress a corresponding detection box. The detection box obtained based on the feature points in the first feature map is called the first detection box. The first detection box is a quadrilateral, which can be represented by the coordinates of the four vertices of the quadrilateral.

[0064] In practical applications, the corresponding detection box can also be regressed for each feature point in the first feature map. However, if a feature point does not correspond to a target object, the regression result of its corresponding detection box can be discarded.

[0065] The first detection box is determined based on convolution operations using a standard convolution kernel, such as a 3x3 matrix. Multiple sampling points obtained from regular grid sampling are used as the objects of the convolution operation. Simply put, a standard 3x3 convolution kernel can be used to perform convolution operations on each group of nine adjacent pixels in the image to obtain feature points in the first feature map. These nine adjacent pixels form the nine pixels that make up the nine-pixel grid. Then, the first detection boxes are obtained based on the first feature map.

[0066] In this embodiment, after obtaining each first detection box based on the standard convolution method, deformable convolution is used to optimize the first feature map, that is, to optimize the feature values ​​of each feature point in the first feature map.

[0067] Specifically, the first feature map is processed by deformable convolution based on the position information of the first detection box to obtain the second feature map, which can be implemented as follows:

[0068] A preset convolution operation is performed on the vertex coordinates of the first detection box to obtain the position coordinates of multiple sampling points. The position information of the first detection box is defined by the vertex coordinates of the first detection box.

[0069] Based on the position coordinates and feature values ​​of each feature point in the first feature map, the feature values ​​corresponding to the multiple sampling points are determined by bilinear interpolation.

[0070] A second feature map is generated based on the position coordinates and feature values ​​of the multiple sampling points.

[0071] In practical applications, performing a preset convolution operation on the vertex coordinates of the first detection box to obtain the position coordinates of multiple sampling points is just one example. Preset convolution operations can also be performed on other position coordinates on the boundary line of the first detection box to obtain the position coordinates of multiple sampling points.

[0072] Here, the preset convolution operation refers to a convolution operation corresponding to a preset convolution kernel size, and the number of sampling points can be preset, such as 9. Taking any first detection box as an example, a 3*3*18 channel convolution operation can be performed on the coordinates of the four vertices of the first detection box to obtain the position coordinates of these nine sampling points. Since each of the nine sampling points has both horizontal and vertical coordinates, resulting in a total of 18 coordinate values, this is 18 channels.

[0073] Combination Figure 2 This example illustrates the execution process of deformable convolution. Figure 2 In this example, assuming the target object is text, the first detection box is the text box. Performing the convolution operation described above on the four vertices of a given first detection box yields the coordinates of nine scattered sampling points. These coordinates are typically decimal values, while the coordinates of each feature point in the first feature map are integers. Therefore, the discrete feature point coordinates in the first feature map are transformed into continuous coordinates, resulting in higher accuracy.

[0074] It should be noted that the vertex coordinates of the first detection box refer to the corresponding pixel positions in the image to be detected. After performing a preset convolution operation on the vertex coordinates of the first detection box in the image to be detected, we obtain the position coordinates of multiple sampling points in the image to be detected. These multiple sampling points refer to multiple sampling points in the first feature map. Based on the mapping relationship between pixel positions in the image to be detected and feature points in the first feature map, we can obtain the position coordinates of the multiple sampling points in the first feature map. In the above steps, "position coordinates of multiple sampling points" refers to the position coordinates of the multiple sampling points in the first feature map. After obtaining the position coordinates of the multiple sampling points, it is necessary to further obtain the feature value corresponding to each sampling point. Specifically, bilinear interpolation can be used to determine the feature value corresponding to each sampling point. Simply put, bilinear interpolation involves: for any given sampling point, identifying the four feature points in the first feature map that are closest to its position coordinates, and performing a weighted summation of the feature values ​​of these four feature points to obtain the feature value corresponding to that sampling point.

[0075] Since these multiple sampling points correspond to a certain feature point in the first feature map, after obtaining the feature values ​​corresponding to each of the multiple sampling points, a convolution operation is performed on these multiple sampling points to obtain a new feature value corresponding to that feature point. This new feature value of the feature point is then used as an element in the second feature map. For example, suppose a first detection box is regressed based on feature point 'a' in the first feature map. A preset convolution operation is performed on the vertex coordinates of this first detection box to obtain the position coordinates of multiple sampling points. Based on the position coordinates and feature values ​​of each feature point in the first feature map, the feature values ​​corresponding to these multiple sampling points are determined by bilinear interpolation. Then, a convolution operation is performed on the feature values ​​of these multiple sampling points to obtain the new feature value corresponding to feature point 'a'.

[0076] The above description only uses one feature point in the first feature map as an example to illustrate the optimization process of its feature value. By performing this optimization process on each feature point in the first feature map, a second feature map composed of the updated feature points can be obtained.

[0077] As can be seen from the above introduction, the optimized feature values ​​of feature points are obtained based on multiple dispersed sampling points, and have a wider receptive field, while the initial feature values ​​of feature points are obtained based on multiple adjacent sampling points. Therefore, the optimized feature values ​​have higher accuracy than the initial feature values.

[0078] Then, based on the optimized second feature map, the final target object detection result is obtained through bounding box regression, which is the second detection box containing the target object in the image to be detected.

[0079] Combination Figure 3aThis example illustrates the difference between the initial detection results and the optimized detection results. Figure 3a In this example, assuming the target object is text, both the first and second detection boxes are text boxes. Let's assume the initial detection result for feature point i in the first feature map is text box q1. After performing deformable convolution with the four vertex coordinates of text box q1 as input to obtain new feature values ​​corresponding to feature point i, text box q1' is regressed based on these new feature values. Text box q1' has a more accurate boundary than text box q1.

[0080] As described above, the second detection box can be considered the final detection result. Compared with the second detection box, the first detection box is only an intermediate result, which is equivalent to optimizing the first detection box into the second detection box.

[0081] To allow users to perceive the optimization effect, the first detection box can be displayed simultaneously with the second detection box to create a comparison.

[0082] Optionally, a first detection box can be displayed in the image to be detected using a first style, and a second detection box can be displayed in the image using a second style. Alternatively, the first detection box can be displayed in the image to be detected using a first style, an update area of ​​the second detection box relative to the first detection box can be determined, and the update area can be displayed in the image using a second style. Optionally, the first style and the second style can be lines of different colors, different shapes, different thicknesses, etc.

[0083] For ease of understanding, combined with Figure 3b Example illustration. For example... Figure 3b As shown, the boundary line of the first detection box is represented by a thinner line. After the update area of ​​the second detection box relative to the first detection box is determined, the boundary line of the update area is represented by a thicker line. The boundary line of the area where the second detection box overlaps with the first detection box is still represented by the aforementioned thinner line.

[0084] In one optional embodiment, it is assumed that the second detection box output based on the second feature map presents a first shape, which is generally a quadrilateral, but not limited thereto. However, in practical applications, the outline of the target object contained in the image to be detected may present a second shape, such as an ellipse, triangle, or circle. In this case, optionally, the shape of the second detection box can be adjusted to the second shape and displayed. For ease of understanding, combined with... Figure 3c Exemplary illustration. In Figure 3cIn this example, assuming the output of the second feature map is the coordinates of the four vertices of the second detection box, the rectangular second detection box, as shown in the diagram, is drawn based on these coordinates. Furthermore, assuming the target object in the image to be detected is a soccer ball (as shown in the diagram), which is circular, the rectangular second detection box can be adjusted to match the outline shape of the target object as a circle. This way, when subsequent applications need to extract the image region of the target object from the image to be detected, the extracted image can contain as little background as possible.

[0085] In addition, in practical applications, users can also determine whether the second detection box displayed in the image is accurate. If the second detection box is found to be inaccurate, the position and boundary of the second detection box can be manually adjusted to match the target object.

[0086] In summary, this embodiment of the invention employs deformable convolutional bounding box optimization to improve the inaccuracy of object detection based on bounding box regression methods. Specifically, based on the initial detection bounding box, new sampling points are learned, and then deformable convolution is used to learn new feature maps to obtain optimized detection bounding boxes. This bounding box optimization process can be performed in parallel for each feature point without reducing the detection speed.

[0087] In practical applications, the target objects to be detected in images vary depending on the application scenario. For example, in text detection, the target object is the region containing the text in the image. In autonomous driving, the image to be detected might be a traffic image captured by a vehicle, and the target objects could be pedestrians, vehicles, etc. Therefore, the subsequent processing based on the output of the detection box positions will differ in different application scenarios. For instance, in the text detection scenario, text detection is generally a prerequisite for text recognition. Therefore, after obtaining the text detection result (i.e., the text box), text recognition processing can continue based on this result. Specifically, the text content within the text box can be extracted based on its vertex coordinates, and then processed such as storage and sensitive word recognition can be performed. Similarly, in autonomous driving, based on the output of the detection box positions corresponding to pedestrians and vehicles, the vehicle can determine where vehicles and pedestrians are ahead, thus providing a basis for driving control decisions.

[0088] Figure 4 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 4 As shown, the method includes the following steps:

[0089] 401. Obtain the image to be detected, which contains the target object.

[0090] 402. Determine the first detection box corresponding to the target object based on the first feature map corresponding to the image.

[0091] 403. Determine the position-aware weight value corresponding to each feature point in the first feature map. The position-aware weight value indicates the distance from the feature point to the boundary of the corresponding detection box.

[0092] 404. Perform deformable convolution processing on the first feature map based on the vertex coordinates of each first detection box to obtain the second feature map.

[0093] 405. Determine the second detection box corresponding to the target object based on the second feature map.

[0094] 406. Based on the position-aware weight values ​​corresponding to each feature point, determine the position-aware weight values ​​corresponding to the vertex coordinates of the second detection box. The position-aware weight values ​​corresponding to the vertex coordinates of the second detection box are used to perform box merging processing on the second detection box.

[0095] 407. Output the merged detection frames.

[0096] As can be understood, as mentioned above, when detecting a target object in an image, multiple detection boxes corresponding to the target object can be regressed based on numerous feature points in the feature map. In this embodiment, the aforementioned first and second detection boxes refer to the various detection boxes output based on the corresponding feature maps. Therefore, the box merging process of the second detection box mentioned in step 406 should be understood as performing box merging processing on a specific second detection box and other overlapping second detection boxes.

[0097] When the image detection method provided in this embodiment of the invention is applied to a text detection scenario, in practice, when the image to be detected contains a certain text block (text content of a certain length), multiple text boxes corresponding to this text block can be obtained based on the feature map. These multiple text boxes correspond to multiple feature points in the feature map, and there will be overlap between these multiple text boxes. These multiple text boxes need to be merged to finally obtain a merged text box corresponding to the text block. Here, the feature map refers to the second feature map obtained after optimizing the initial first feature map corresponding to the image to be detected; correspondingly, the multiple text boxes refer to the multiple second text boxes obtained based on the second feature map.

[0098] In text detection scenarios, the multiple first text boxes obtained based on the first feature map corresponding to the image to be detected often have inaccurate boundaries, especially when encountering long text blocks, where the detection results for the text boxes corresponding to these long text blocks become even more inaccurate. However, using an optimized second feature map can improve the accuracy of text box detection results. But a single text block will regress multiple overlapping text boxes, requiring box merging to obtain a more accurate text box corresponding to that text block—the merged text box.

[0099] This embodiment provides a method for box merging. The execution process of this box merging method is illustrated below using a text detection scenario as an example.

[0100] The box merging method provided in this embodiment can be used to merge multiple overlapping text boxes corresponding to a long text block into a complete long text box. In summary, while predicting each text box, a score for that text box is also predicted as a weight in the box merging process. This score is the position-aware weight value corresponding to the corresponding feature point. Since text box prediction is performed for each feature point in the feature map, that is, a corresponding position-aware weight value is predicted for each feature point, the box merging process is finally achieved based on the position-aware weight values ​​corresponding to each feature point in the feature map.

[0101] Optionally, the prediction of the location-aware weight values ​​can be performed on either the first feature map or the second feature map.

[0102] Assuming the process is performed on the first feature map, a neural network model can be pre-trained. This model possesses text detection capabilities and the ability to learn position-aware weights for feature points. When the image to be detected is input into this neural network model, it extracts the first feature map and, based on this map, performs prediction processing for the first text boxes corresponding to each feature point. Simultaneously, for each feature point, it outputs the corresponding position-aware weight value.

[0103] The position-aware weight value reflects the distance of the feature point to the text box boundary. Taking the left and right boundaries as an example, if a feature point is closer to the left boundary of the text box (high weight), then its distance to the right boundary will be farther (low weight).

[0104] It's important to clarify that this position-aware weight value reflects the distance from the feature point to the text box boundary. First, strictly speaking, this distance refers to the distance from the pixel location of the feature point in the image to be detected to the text box boundary, since the text box is a concept within the coordinate system of the image to be detected. Second, the "text box" here should be understood as the distance from the feature point to the boundary of the "ideal text box," not the distance to the boundary of the "actually predicted text box." The "ideal text box" can be considered as the text box used for supervisory information annotation. Figure 5 For example, suppose an image contains the text block "sales revenue". During model training, it will be labeled... Figure 5 The text box K1 shown in the diagram serves as supervision information. After obtaining the first feature map through feature extraction, it is assumed that text boxes r1 and r2 are predicted based on feature points i and j in the first feature map, respectively. It is also assumed that these two feature points are feature points corresponding to the text block "sales amount". Then, when predicting the position perception weight values ​​corresponding to feature points i and j, it is to predict the distance scores of feature points i and j to the boundary of text box K1.

[0105] The above example using the first feature map illustrates the process of predicting the position-aware weight values ​​corresponding to each feature point in the first feature map. In fact, the process of predicting the position-aware weight values ​​corresponding to each feature point in the second feature map is similar. As mentioned earlier, the difference between the first and second feature maps lies only in the feature values ​​of the feature points; the position coordinates of the feature points are the same. The difference in feature values ​​has a significant impact on the prediction results of the text box boundaries, but little impact on the position-aware weight values ​​of the feature points. Therefore, it is possible to predict the position-aware weight values ​​of feature points using either the first or second feature map.

[0106] In this embodiment, it is assumed that the position-aware weight values ​​of feature points are predicted for the first feature map. Then, after feature extraction of the image to be detected to obtain the first feature map, text boxes and position-aware weight values ​​are predicted for each feature point in the first feature map, thereby obtaining the first text box and position-aware weight value corresponding to each feature point in the first feature map.

[0107] For the coordinates of the four vertices of each first text box, deformable convolution is performed to obtain new feature values ​​corresponding to the feature points. Finally, the second feature map is obtained from the new feature values ​​corresponding to all feature points. The processing procedure of deformable convolution is described in the relevant description of the previous embodiment, and will not be repeated here.

[0108] Next, text box prediction is performed on each feature point in the second feature map to obtain the second text box corresponding to each feature point. In this way, multiple second text boxes will be obtained based on the second feature map. Some of these second text boxes will overlap. The overlapping text boxes often correspond to the same text block and need to be merged.

[0109] Taking a second text box as an example, to perform box merging, it is first necessary to determine the position-aware weight values ​​corresponding to the vertex coordinates of this second text box. Specifically, based on the position-aware weight values ​​corresponding to each feature point in the first or second feature map, the position-aware weight values ​​corresponding to the vertex coordinates of the second text box can be determined by bilinear interpolation.

[0110] Specifically, in practical applications, the coordinates of the four vertices of the second text box refer to the corresponding pixel positions in the image to be detected. Based on the mapping relationship between the feature point positions in the first feature map and the pixel positions in the image to be detected, the position coordinates of the four vertices of the second text box in the first feature map can be determined. These position coordinates are in decimal form. The coordinate values ​​of each feature point in the first feature map (taking the determination of the position-aware weight values ​​corresponding to each feature point in the first feature map as an example) are in integer form. For the position coordinates (let's say L1) of a certain vertex coordinate in the second text box corresponding to the position coordinate in the first feature map (let's say L1'), we can first determine the four feature points in the first feature map that are closest to the position coordinate L1'. Then, based on the position-aware weight values ​​corresponding to these four feature points, we determine the position-aware weight value corresponding to the position coordinate L1' by weighted summation. This position-aware weight value is then used as the position-aware weight value corresponding to the vertex coordinate L1 of the second text box. The same logic applies to the other vertex coordinates of the second text box.

[0111] For each second text box obtained based on the second feature map, the position-aware weight value corresponding to the vertex coordinates is determined using the method described above. Then, box merging can be performed based on the position-aware weight values ​​corresponding to the vertex coordinates of each second text box.

[0112] The box merging process can be implemented as follows:

[0113] Identify at least two second text boxes whose overlapping area is greater than a set threshold;

[0114] Determine the position-aware weight value corresponding to the vertex coordinates of each of the at least two second text boxes;

[0115] The at least two second text boxes are merged according to the following rules to obtain a merged text box: For the vertex coordinates of the at least two second text boxes in the same direction, the coordinates are weighted and summed according to their respective position-aware weight values ​​to obtain the merged coordinates corresponding to that direction.

[0116] For ease of description and understanding, combined with Figure 6 The process of merging boxes is illustrated by example.

[0117] exist Figure 6 In this context, assume that two second text boxes, text box D1 and text box D2, are determined based on feature points p1 and p2 in the second feature map. The coordinates of the four vertices of text box D1 are represented as V11, V12, V13, and V14, and the coordinates of the four vertices of text box D2 are represented as V21, V22, V23, and V24. Figure 6 As shown in the illustration, it is assumed that the vertex coordinates of the two text boxes corresponding to the top left corner are V11 and V21; the vertex coordinates of the two text boxes corresponding to the top right corner are V12 and V22; the vertex coordinates of the two text boxes corresponding to the bottom left corner are V13 and V23; and the vertex coordinates of the two text boxes corresponding to the bottom right corner are V14 and V24.

[0118] After obtaining the position-aware weight values ​​corresponding to the coordinates of the eight vertices mentioned above, assuming the position-aware weight value for V11 is a1 and the position-aware weight value for V21 is a2, and a1 is greater than a2, then the merged result of the coordinates of these two top-left vertices can be expressed as: V11*a1 + V21*a2. This can be understood as follows: assuming the x-coordinate contained in V11 is X11 and the x-coordinate contained in V21 is X21, the weighted sum of X11*a1 + X21*a2 is represented as Xa, and Xa is the merged x-coordinate. The y-coordinate is similar, and the merging process for the other top-right, bottom-left, and bottom-right vertices is also similar.

[0119] As the above example shows, vertices with larger position-aware weight values ​​have a more significant impact on the position of the merged text box because vertices with larger position-aware weight values ​​are closer to the ideal text box. Figure 6 In this example, we assume that the merged text box D3 is obtained using the above merging method. Finally, we output the merged text box.

[0120] As illustrated in the previous embodiments regarding the output methods of the second and first detection boxes, similarly, the merged text boxes can be displayed in different styles in the image for the user to view the merging effect. If the user finds that the merging result is inaccurate (e.g., it does not correspond well to the boundaries of the text blocks), they can manually adjust it. Optionally, only the merged text boxes can be displayed in the image, without displaying the first and second text boxes.

[0121] As can be seen from the above example, in this embodiment, by automatically predicting the position-aware weight value corresponding to each feature point, the essential purpose of this position-aware weight value is to allow feature points near the edge to guide the merging of the edges of the text box corresponding to a text block, thus obtaining a text detection box with more accurate boundaries. This merging method is relatively simple and efficient.

[0122] As mentioned above, the image detection method provided by this invention can be executed in the cloud, where multiple computing nodes can be deployed, each with processing resources such as computing and storage. In the cloud, multiple computing nodes can be organized to provide a certain service; of course, a single computing node can also provide one or more services. The cloud can provide this service by providing an external service interface, which users can call to use the corresponding service. Service interfaces include Software Development Kits (SDKs), Application Programming Interfaces (APIs), and other forms.

[0123] According to the solution provided in this embodiment of the invention, the cloud can provide a service interface for image detection services. Users invoke this image detection service interface through their user devices to trigger a request to the cloud to invoke the image detection service interface. The cloud determines the computing node that responds to the request and utilizes the processing resources in that computing node to perform the following steps:

[0124] Receive a request from a user equipment to call the image detection service interface, the request including an image to be detected, the image containing a target object;

[0125] The following steps are performed using the processing resources corresponding to the image detection service interface:

[0126] A first detection box corresponding to the target object is determined based on the first feature map corresponding to the image;

[0127] Based on the position information of the first detection box, deformable convolution processing is performed on the first feature map to obtain the second feature map;

[0128] The second detection box corresponding to the target object is determined based on the second feature map;

[0129] The position information of the second detection frame is output to the user equipment.

[0130] The detailed process of the image detection service interface performing image detection processing using processing resources can be found in the relevant descriptions in the other embodiments mentioned above, and will not be repeated here.

[0131] For ease of understanding, combined with Figure 7 To illustrate with an example. In Figure 7 In this context, when a user wants to perform object detection processing on an image to detect target objects, the user device E1 calls the image detection service interface to send a call request to the cloud computing node E2. This call request includes the image to be detected and may also include category information corresponding to the target object instances to be detected. Figure 7 The illustrated method for calling the image detection service interface is as follows: A user uses a specific app with an "Upload" button on one of its interfaces. The user loads the image to be detected on this interface and clicks the upload button, triggering the aforementioned call request. In other words, this app is a client program providing image detection services in the cloud, and the upload button within this program is the application programming interface (API) for calling this service. After loading the original image to be detected, the user can also perform various image editing operations on the image using the various image editing tools provided under the "Image Editing" menu, such as scaling, cropping, and other preprocessing, to enhance image quality.

[0132] In this embodiment, it is assumed that after receiving the call request, the cloud computing node E2 determines, based on the aforementioned information, which type of target object needs to be detected in the image to be detected, and then performs the detection process. The detection process is described in the previous embodiment and will not be repeated here. Afterwards, the cloud computing node E2 sends the final detection result—the location information of the second detection box corresponding to the target object—to the user device E1. The user device E1 then displays the image to be detected marked with the second detection box on the interface based on the location information.

[0133] When a user finds that the position of the second detection box does not match the actual location of the target object in the image to be detected, the user can manually adjust the position of the second detection box and feed back the adjustment information to the computing node E2 through the user device E1 so that the computing node E2 can optimize the model accordingly.

[0134] Alternatively, one or more operation items can be displayed on the user device E1 for subsequent processing based on the detection box marking results, such as the adjustment-related operation items mentioned above. In addition, taking a text detection scenario as an example, some operation items corresponding to applications related to the text detection scenario can also be included, such as... Figure 7The diagram illustrates operations such as text recognition. Text recognition refers to extracting a corresponding image region from the image to be detected based on the position of the detection box (i.e., the text box). This image region is the area containing the text content. Then, this image region is input into the text recognition model to obtain the text recognition result.

[0135] In practical applications, text detection is required in many fields, and the technical solutions of the embodiments of the present invention can be used in all of them. The following are examples of several embodiments.

[0136] Figure 8 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 8 As shown, the method may include the following steps:

[0137] 801. Obtain product images.

[0138] 802. Determine the first text box from the product image based on the first feature map corresponding to the product image.

[0139] 803. Perform deformable convolution processing on the first feature map based on the position information of the first text box to obtain the second feature map.

[0140] 804. Determine the second text box from the product image based on the second feature map.

[0141] 805. Detect and process the text content extracted from the second text box.

[0142] The solution provided in this embodiment is applicable to e-commerce scenarios, where merchants upload product images containing a lot of text, such as product names and descriptions. Text detection and recognition can be performed on the product images to extract the text content. Text recognition is based on determining the location of text blocks within the product image through text detection. A text block consists of multiple adjacent characters.

[0143] The text detection process can be implemented using the solution provided in this embodiment. For details on the specific implementation process of the above solution, please refer to the relevant descriptions in the foregoing embodiments; they will not be repeated here.

[0144] It should be noted that this embodiment does not illustrate the process of merging overlapping second text boxes after obtaining each second text box. In fact, when the detection result shows that there are overlapping second text boxes, the box merging processing method provided in the aforementioned embodiment can be used to perform box merging processing. At this time, the positions of the merged multiple boxes will be marked in the product image.

[0145] For ease of description, we assume here that the second text boxes obtained based on the second feature map are the final detection results. Then, based on the position of each second text box, the product image can be cropped to obtain the image region corresponding to each second text box. Each second text box corresponds to one text block. Next, each image region is input into the text recognition model, which outputs the text content contained in each image region.

[0146] Afterwards, the extracted text content can be processed according to different application purposes.

[0147] For example, e-commerce platforms need to review whether the text content meets requirements, such as whether it contains sensitive words. In practical applications, a sensitive word database can be pre-built. If words in the sensitive word database are identified in the product image, the product image is considered unsuitable for publication, and the merchant is given a corresponding warning message.

[0148] For example, the category of a product image can be determined based on keywords contained in the text content. The recognized text content may contain product descriptions and other identifying information such as product names. If pre-defined keywords for category classification can be extracted from this information, such as "shoes," "hats," and "skirts," then product category classification can be achieved based on the extracted keywords.

[0149] Figure 9 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 9 As shown, the method may include the following steps:

[0150] 901. Extract features from the teaching images to obtain the first feature map.

[0151] 902. Determine the first text box based on the first feature map.

[0152] 903. Perform deformable convolution on the first feature map based on the vertex coordinates of the first text box to obtain the second feature map.

[0153] 904. Determine the second text box based on the second feature map.

[0154] 905. Perform teaching image search processing based on the text content extracted from the second text box.

[0155] The solution provided in this embodiment can be applied to educational scenarios. In educational scenarios, teachers may use presentation tools such as blackboards and PPTs during the teaching process. Students can take pictures of the teaching images using these presentation tools. When students take a large number of teaching images, they face the need to classify, organize, and retrieve these images as needed.

[0156] When a student needs to search for teaching images related to a specific knowledge point from a large number of collected teaching images, the text detection scheme provided in this embodiment of the invention can be used to perform text detection processing on each collected teaching image. First, text boxes are detected in each teaching image. Then, text recognition processing is performed on the text content contained in each text box to obtain the text content. Afterwards, using the name of the desired knowledge point as the search keyword and the text content recognized in each teaching image as the search database, teaching images containing that knowledge point can be searched.

[0157] In educational settings, besides the aforementioned need for text detection on teaching images, text detection processing can also be performed on student homework, test papers, and other teaching materials. For example, if parents want to collect numerous test questions for summarizing and using as a reference when creating tests for their children, they can photograph their children's homework, test papers, and other materials to obtain the corresponding images, or they can collect images of homework and test papers online. Then, using the detection scheme described above, various text boxes can be detected in the images. The text boxes corresponding to the test questions are then filtered out, and text recognition processing is performed on the image regions corresponding to these text boxes to obtain the text content contained within, i.e., the test question content.

[0158] Figure 10 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 10 As shown, the method may include the following steps:

[0159] 1001. Extract features from the medical record image to obtain the first feature map.

[0160] 1002. Determine the first text box based on the first feature map.

[0161] 1003. Perform deformable convolution processing on the first feature map based on the vertex coordinates of the first text box to obtain the second feature map.

[0162] 1004. Determine the second text box based on the second feature map.

[0163] 1005. Perform medical record image search processing based on the text content extracted from the second text box.

[0164] The solution provided in this embodiment can be applied to medical scenarios, where a large number of medical record images and medical images (such as various contrast images) can be generated, and text detection processing can be performed on these images.

[0165] For example, when an institution needs to perform statistical analysis of medical records, it can use the text detection scheme provided in this embodiment of the invention to perform text detection processing on the collected medical record images. First, it detects the text boxes contained in each medical record image. Then, it performs text recognition processing on the text content contained in each text box to obtain the text content. Afterwards, based on the text content corresponding to each medical record image, it searches for medical record images that match set keywords, such as a certain disease, time period, etc.

[0166] The text detection process for medical record images and medical images can be implemented by referring to the detection process described in the other embodiments above, and will not be repeated here.

[0167] The above examples illustrate the application scenarios to which the image detection scheme provided in this embodiment of the invention can be applied, but in reality, it is not limited to these examples.

[0168] The image detection apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these image detection apparatuses can all be configured using commercially available hardware components through the steps taught in this solution.

[0169] Figure 11 This is a schematic diagram of the structure of an image detection device provided in an embodiment of the present invention, as shown below. Figure 11 As shown, the device includes: an acquisition module 11, a detection module 12, and an output module 13.

[0170] The acquisition module 11 is used to acquire an image to be detected, wherein the image contains a target object.

[0171] The detection module 12 is configured to determine a first detection box corresponding to the target object based on a first feature map corresponding to the image; and to perform deformable convolution processing on the first feature map based on the position information of the first detection box to obtain a second feature map, and to determine a second detection box corresponding to the target object based on the second feature map.

[0172] Output module 13 is used to output the second detection frame.

[0173] Optionally, the output module 13 may be specifically used to: display the first detection box in the image in a first style; and display the second detection box in the image in a second style, or determine the update area of ​​the second detection box relative to the first detection box and display the update area in the image in a second style.

[0174] Optionally, the second detection box presents a first shape, and the output module 13 can be specifically used to: adjust the shape of the second detection box to the second shape if the outline of the target object presents a second shape; and display the second detection box presenting the second shape in the image.

[0175] Optionally, the detection module 12 is specifically used to: perform a preset convolution operation on the vertex coordinates of the first detection box to obtain the position coordinates of multiple sampling points, wherein the position information of the first detection box is defined by the vertex coordinates of the first detection box; determine the feature values ​​corresponding to the multiple sampling points by bilinear interpolation based on the position coordinates and feature values ​​of each feature point in the first feature map; and generate a second feature map based on the position coordinates and feature values ​​of the multiple sampling points.

[0176] Optionally, the device further includes: a merging processing module, configured to determine the position-aware weight value corresponding to each feature point in the first feature map or the second feature map, wherein the position-aware weight value indicates the distance from the feature point to the boundary of the corresponding detection box; and, based on the position-aware weight value corresponding to each feature point, determine the position-aware weight value corresponding to the vertex coordinates of the second detection box, wherein the position-aware weight value corresponding to the vertex coordinates of the second detection box is used to perform box merging processing on the second detection box.

[0177] Optionally, the merging processing module is specifically used to: determine at least two second text boxes with an overlapping area greater than a set threshold; determine the position-aware weight value corresponding to the vertex coordinates of the at least two second text boxes; and perform frame merging processing on the at least two second text boxes according to the following rules to obtain merged text boxes: for the vertex coordinates of the at least two second text boxes in the same direction, perform a weighted summation operation on the coordinates according to their respective position-aware weight values ​​to obtain the merged coordinates corresponding to that direction.

[0178] Figure 11 The device shown can perform the image detection method provided in the foregoing embodiments. For details of the execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0179] In one possible design, the above Figure 11 The structure of the image detection device shown can be implemented as an electronic device, such as... Figure 12 As shown, the electronic device may include a processor 21 and a memory 22. The memory 22 stores executable code, which, when executed by the processor 21, enables the processor 21 to at least implement the image detection method provided in the foregoing embodiments.

[0180] Optionally, the electronic device may also include a communication interface 23 for communicating with other devices.

[0181] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image detection method provided in the foregoing embodiments.

[0182] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image detection method, characterized in that, include: Acquire an image to be detected, wherein the image contains the target object; A first detection box corresponding to the target object is determined based on the first feature map corresponding to the image; Based on the position information of the first detection box, deformable convolution processing is performed on the first feature map to obtain the second feature map; The second detection box corresponding to the target object is determined based on the second feature map; Determine the position-aware weight value corresponding to each feature point in the first feature map or the second feature map, wherein the position-aware weight value indicates the distance from the feature point to the boundary of the corresponding detection box; Based on the position-aware weight values ​​corresponding to each feature point, determine the position-aware weight values ​​corresponding to the vertex coordinates of the second detection box; Based on the position perception weight values ​​corresponding to the vertex coordinates of the second detection box, the second detection box is merged. The second detection box after the output box is merged.

2. The method according to claim 1, characterized in that, The method further includes: displaying the first detection box in the image in a first style; The output of the second detection box includes: The second detection box is displayed in the image in a second style, or, Determine the update region of the second detection box relative to the first detection box, and display the update region in the image in a second style.

3. The method according to claim 1, characterized in that, The second detection box takes the shape of the first shape, and the output of the second detection box includes: If the outline of the target object presents a second shape, then the shape of the second detection box is adjusted to the second shape; A second detection box, which presents the second shape, is displayed in the image.

4. The method according to claim 1, characterized in that, The method further includes: Based on the position information of the second detection box, extract the target object contained within the second detection box; The target object is then processed.

5. An image detection method, characterized in that, include: Receive a request from a user equipment to call the image detection service interface, the request including an image to be detected, the image containing a target object; The following steps are performed using the processing resources corresponding to the image detection service interface: A first detection box corresponding to the target object is determined based on the first feature map corresponding to the image; Based on the position information of the first detection box, deformable convolution processing is performed on the first feature map to obtain the second feature map; The second detection box corresponding to the target object is determined based on the second feature map; Determine the position-aware weight value corresponding to each feature point in the first feature map or the second feature map, wherein the position-aware weight value indicates the distance from the feature point to the boundary of the corresponding detection box; Based on the position-aware weight values ​​corresponding to each feature point, determine the position-aware weight values ​​corresponding to the vertex coordinates of the second detection box; Based on the position perception weight values ​​corresponding to the vertex coordinates of the second detection box, the second detection box is merged. The position information of the second detection frame after frame merging is output to the user equipment.

6. An image detection method, characterized in that, include: Get product images; The first text box is determined from the product image based on the first feature map corresponding to the product image; The first feature map is subjected to deformable convolution processing based on the position information of the first text box to obtain the second feature map; The second text box is determined from the product image based on the second feature map; Determine the position-aware weight value corresponding to each feature point in the first feature map or the second feature map, wherein the position-aware weight value indicates the distance from the feature point to the boundary of the corresponding detection box; Based on the position-aware weight values ​​corresponding to each feature point, determine the position-aware weight values ​​corresponding to the vertex coordinates of the second text box; Based on the position-aware weight values ​​corresponding to the vertex coordinates of the second text box, the second text box is merged. The text content extracted from the second text box after the box merging process is detected and processed.

7. The method according to claim 6, characterized in that, The detection and processing of the text content extracted from the second text box includes: Determine whether the text content contains any set sensitive words; or, Based on the keywords contained in the text content, the category of the product corresponding to the product image is determined.

8. An image detection device, characterized in that, include: The acquisition module is used to acquire an image to be detected, wherein the image contains the target object; The detection module is used to determine the first detection box corresponding to the target object based on the first feature map corresponding to the image; Furthermore, based on the position information of the first detection box, deformable convolution processing is performed on the first feature map to obtain a second feature map; the second detection box corresponding to the target object is determined based on the second feature map; the position-aware weight value corresponding to each feature point in the first feature map or the second feature map is determined, the position-aware weight value indicating the distance from the feature point to the boundary of the corresponding detection box; based on the position-aware weight value corresponding to each feature point, the position-aware weight value corresponding to each vertex coordinate of the second detection box is determined; and the second detection box is merged based on the position-aware weight value corresponding to each vertex coordinate of the second detection box. The output module is used to output the second detection box after the box merging process.

9. An electronic device, characterized in that, include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the image detection method as described in any one of claims 1 to 4.

10. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the image detection method as described in any one of claims 1 to 4.

Images