A target detection method, terminal device and storage medium

Abstract

The present application is applicable to the field of neural network technology, and provides a target detection method, a terminal device and a storage medium. The method comprises: using an improved Faster RCNN as a target detection model, during training, using an RPN network to use a convex hull method to find the relative order of four target points, and then using a forward and backward cyclic shift method to obtain multiple groups of orders of the four target points, and determining one of the groups of orders as a correction order of the step according to the multiple groups of orders of the four points and the minimum circumscribed rectangle of the four target points, thereby calculating a loss function based on the coordinates of the four target points under the correction order to update the parameters in the target detection model; in the prediction stage, using an RPN network to predict the correction order of the four target points based on a determined cyclic shift method, thereby obtaining a detection frame based on the correction order, and this method can solve the problem of oscillation when an oblique frame is used as a detection frame during target detection.

Description

Technical Field

[0001] The present application belongs to the field of neural network technology, and in particular, relates to a target detection method, a terminal device and a storage medium. Background Art

[0002] With the development of artificial intelligence, the accuracy requirements for target detection models in images are getting higher and higher. Usually, when performing target detection, candidate regions are first generated on the image, and then the candidate regions are adjusted to obtain the final target detection frame. At present, most detection frames are in the form of positive frames, that is, the four frames of the detection frame are parallel to the two borders of the image.

[0003] In practical applications, the size and shape of objects in images are not fixed, and some objects even have special arrangements, such as vehicles, airplanes, and ships placed in rows at an angle. This makes the straight frame unsuitable for such object detection, while the oblique frame may be more suitable for such object detection. However, the oblique frame often has the problem of misalignment of the order of the four points, which causes oscillation during training and affects the detection accuracy of the trained model. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a target detection method, a terminal device and a storage medium to solve the problem that when an oblique frame is used as a target detection frame, the training process generates oscillations, which affects the detection accuracy of the trained model.

[0005] A first aspect of an embodiment of the present application provides a target detection method, which is applied to a target detection model including an RPN network. The method includes:

[0006] Input a first image into a target detection model, and output a first image including at least one first oblique frame, wherein the first image is an image in a training set, and the RPN network is used to generate four first target points with an initial order on the first image, and obtain a target order of the four target points based on the first coordinates of the four first target points and the minimum circumscribed rectangle of the four first target points, and the four first target points with the target order are used to generate a first oblique frame;

[0007] According to the distance deviation between the second coordinates of the four vertices of the first oblique frame and the label coordinates of the label points calibrated on the first image, the parameters in the target detection model are reversely updated to obtain the trained target detection model;

[0008] The second image is input into the trained target detection model, and a second image containing at least one second oblique frame is output, wherein the second image is an image to be detected, and the second oblique frame contains the target on the second image. The RPN network is used to generate four second target points with an initial order on the second image, and a target order of the four second target points is obtained based on the initial order and the order deviation of the four second target points, wherein the order deviation is the deviation between the target order of the four first target points and the initial order of the four first target points, and the target order of the four second target points with the target order is used to generate the second oblique frame.

[0009] A second aspect of an embodiment of the present application provides a target detection device, which is applied to a target detection model including an RPN network. The device includes:

[0010] A training unit is used to input a first image into a target detection model, and output a first image including at least one first oblique frame, wherein the first image is an image in a training set, and the RPN network is used to generate four first target points with an initial order on the first image, and obtain a target order of the four target points based on first coordinates of the four first target points and a minimum circumscribed rectangle of the four first target points, and the four first target points with the target order are used to generate a first oblique frame;

[0011] An updating unit, configured to reversely update the parameters in the target detection model according to the distance deviation between the second coordinates of the four vertices of the first oblique frame and the label coordinates of the label points calibrated on the first image, so as to obtain a trained target detection model;

[0012] A detection unit is used to input a second image into a trained target detection model, and output a second image containing at least one second oblique frame, wherein the second image is an image to be detected, and the second oblique frame contains the target on the second image. The RPN network is used to generate four second target points with an initial order on the second image, and based on the initial order and the order deviation of the four second target points, a target order of the four second target points is obtained, wherein the order deviation is the deviation between the target order of the four first target points and the initial order of the four first target points, and the target order of the four second target points with the target order is used to generate the second oblique frame.

[0013] The third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method provided in the first aspect of the embodiment of the present application are implemented.

[0014] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the steps of the method provided in the first aspect of the embodiments of the present application are implemented.

[0015] The fifth aspect of the embodiments of the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method provided in the first aspect of the embodiments of the present application.

[0016] The embodiment of the present application provides a target detection method, which is based on an improved FasterRCNN as a target detection model. During training, the convex hull method is used to find four target points with an initial order on the training image through the RPN network, and the target order of the four points is obtained according to the coordinates of the four points and the minimum circumscribed rectangle of the four points. Then, the oblique frame during training is obtained according to the target order of the four points; according to the deviation between the oblique frame during training and the label coordinates of the image, the target detection model is updated in reverse; during detection, the convex hull method is also used to find four target points with an initial order on the detection image through the RPN network, and then according to the order deviation between the target order and the initial order during the training image, the initial order of the four target points on the detection image is updated to the target order, and finally, according to the target order, a detection oblique frame containing the target is generated on the detection image. Since the order relationship between the initial order and the target order is determined during training, the problem of oscillation when the oblique frame is used as the detection frame during target detection is solved, and at the same time, the accuracy of the target order obtained in the detection stage is improved, thereby improving the detection accuracy.

[0017] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a schematic diagram of the structure of a target detection model provided in an embodiment of the present application;

[0020] Figure 2 It is a structural diagram of a feature extraction network provided in an embodiment of the present application;

[0021] Figure 3 It is a structural diagram of a Swin Transformer Block provided in an embodiment of the present application;

[0022] Figure 4is a schematic diagram of a method for calculating distance deviation provided in an embodiment of the present application;

[0023] Figure 5 It is a flowchart of the training process of the target detection model provided in the embodiment of the present application;

[0024] Figure 6 It is a flowchart of the detection process of the target detection model provided in the embodiment of the present application;

[0025] Figure 7 is a grayscale schematic diagram of a second image before detection by the target detection model provided in an embodiment of the present application;

[0026] Figure 8 The target detection model provided in the embodiment of the present application is Figure 7 A grayscale schematic diagram of the second image after detection is shown;

[0027] Fig. 9 is a grayscale schematic diagram of another second image before detection by the target detection model provided in an embodiment of the present application;

[0028] Fig.10 The target detection model provided in the embodiment of the present application is Fig. 9 A grayscale schematic diagram of the second image after detection is shown;

[0029] Fig.11 It is a schematic block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0031] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0032] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0033] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0035] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0036] An embodiment of the present application provides a neural network model, which, after training, can be used to detect oblique frame targets. Therefore, the neural network model can be called a target detection model.

[0037] See also Figure 1 , the target detection model provided in the embodiment of the present application adopts the Faster RCNN model as the model framework. That is, the target detection model includes: a feature extraction network, an RPN (region proposal Network) network, a ROI Align network and a regression network connected in sequence, wherein the output of the feature extraction network also needs to be used as the input of the ROI Align network. That is, the input of the ROI Align network includes the output of the feature extraction network and the output of the RPN network.

[0038] In order to make the final detection box an oblique box, the RPN network adopts the ORN (oriented Response Networks) algorithm.

[0039] Of course, in practical applications, the regression network also has a classification function, for example, obtaining the category of the target in the detection frame. Therefore, the regression network can also be named a classification regression network. In addition, the ROI Align network can also be a ROIPooling network. The embodiment of the present application does not limit this.

[0040] The embodiment of the present application focuses on improving the feature extraction network and RPN network in the Faster RCNN framework. Other networks can be replaced by networks with the same functions, and the embodiment of the present application does not limit this. The following will focus on describing the feature extraction network and the RPN network.

[0041] The feature extraction network may adopt a Swin Transformer network, which includes sequentially connected patch partition layers and N consecutive stage extraction networks.

[0042] See also Figure 2 , the embodiment of the present application can use 4 stages to extract the network (for example, stage 1 to stage 4). Of course, in practical applications, more or less than 4 stages can also be used to extract the network.

[0043] The block layer is used to divide the image into different image blocks according to a certain size. For example, the image can be divided into 4×4 image blocks.

[0044] See also Figure 2 , the first stage extraction network includes sequentially connected Linear Embedding layers and at least two Swin Transformer Blocks. The subsequent N-1 stage extraction networks include sequentially connected Patch Merging layers and at least two Swin Transformer Blocks.

[0045] Among them, linear embedding is used to compress the dimension of the data to be processed. The Patch Merging layer is used to reduce the size of the data to be processed.

[0046] It can be understood from the above examples that the Swin Transformer network as a feature extraction network includes multiple Swin Transformer Blocks.

[0047] As another embodiment of the present application, see Figure 2 , the feature extraction network includes 4 stage extraction networks. The first stage extraction network may include 2 Swin Transformer Blocks. The second stage extraction network may include 2 Swin Transformer Blocks. The third stage extraction network may include 6 Swin Transformer Blocks. The fourth stage extraction network may include 2 Swin Transformer Blocks.

[0048] See also Figure 3, Swin Transformer Block includes two connected sub-units, where the first sub-unit includes: LN layer (layernorm), W-MSA layer (windowbased multi-head selfattention), LN layer (layernorm) and MLP layer (Multi-Layer Perceptron) connected in sequence. The second sub-unit includes LN layer, SW-MSA layer (shifted window-based multi-head self attention), LN layer and MLP layer connected in sequence.

[0049] The first sub-unit divides the image into different blocks of a certain size, and each time the transformer's attention is calculated only within the image block. Of course, this will result in the receptive field of each pixel not being improved. The second sub-unit can change the way the image blocks are divided, so that the image blocks for each pixel to perform attention calculations are in a state of change, which can improve the receptive field of the pixel.

[0050] Combination Figure 3 , if the input of the first LN layer in the first subunit is denoted by z l-1 , then the input of the second LN layer in the first subunit is recorded as but The output of the first subunit is denoted as z l ,but

[0051] The input of the first LN layer in the second subunit is z l , then the input of the second LN layer in the second subunit is recorded as but The output of the second subunit is denoted as z l+1 ,but

[0052] From the diagram, we can understand that after each stage of network extraction, the image depth becomes 1 / 4 of the original, and the number of channels becomes 4 times the original. Because after MLP, the number of channels is reduced by half, that is, after each stage of network extraction, the total amount of data becomes 1 / 2 of the original.

[0053] The RPN network provided in the embodiment of the present application uses the ORN algorithm to generate oblique frames.

[0054] In the training stage, when the RPN network processes an image, a 4-point convex hull method may be used to obtain an initial order of the 4 target points, where the initial order of the 4 target points includes a relative order of the 4 target points.

[0055] As an example, the four target points are point 1, point 2, point 3, and point 4. Of course, each point has coordinates; for example, the coordinates of point 1 are (x1, y1), the coordinates of point 2 are (x2, y2), the coordinates of point 3 are (x3, y3), and the coordinates of point 4 are (x4, y4). The initial order of the four target points is: 1234.

[0056] The relative order of the four target points is: the previous target point is in front of the next target point in the initial order, and the last target point is in front of the first target point. That is, the relative order of the four target points is: 1 is in front of 2, 2 is in front of 3, 3 is in front of 4, and 4 is in front of 1.

[0057] According to the relative order of the four target points, at least four groups of absolute orders can be obtained, for example, 1234; 2341; 3412; 4123. The embodiment of the present application can adopt 4123; 1234; 2341 as the absolute order of the four points.

[0058] In practical applications, the label points on the image have a unique order. Therefore, it is necessary to select a set of orders from the determined absolute order to correspond to the unique order of the label points on the image.

[0059] In the embodiment of the present application, the minimum bounding rectangle of the four points can be obtained according to the coordinates of the four points, and then the distance deviation between the four coordinate points and the minimum bounding rectangle in different absolute orders is calculated.

[0060] As an example, see Figure 4 , taking the absolute order 4123 as an example, the first deviation d1 of the first point is obtained by subtracting the projection value (x4) of the first point 4 on the x-axis from the projection value (x0) of the center point of the minimum circumscribed rectangle on the x-axis;

[0061] The projection value (y1) of the second point 1 in the absolute order on the y-axis minus the projection value (y0) of the center point of the minimum circumscribed rectangle on the y-axis obtains a second deviation d2 of the second point 1;

[0062] The projection value (x2) of the third point 2 in the absolute order on the x-axis is subtracted from the projection value (x0) of the center point of the minimum circumscribed rectangle on the x-axis to obtain a third deviation d3 of the third point 2;

[0063] Subtract the projection value (y0) of the center point of the minimum circumscribed rectangle on the y-axis from the projection value (y3) of the fourth point 3 in the absolute order on the y-axis to obtain a fourth deviation d4 of the fourth point 3;

[0064] The sum of the absolute value of the first deviation, the absolute value of the second deviation, the absolute value of the third deviation and the absolute value of the fourth deviation is calculated to obtain the distance deviation between the four points and the minimum circumscribed rectangle in the absolute order.

[0065] The coordinates of a set of absolute order corresponding to the smallest distance deviation are output to the next network (ROI Align network).

[0066] Of course, the set of absolute orders with the smallest distance deviation is obtained by cyclic shifting the initial order. Therefore, it is necessary to record the cyclic shifting method so that in the target detection stage, the coordinate order of the points input to the next network (ROI Align network) can be calculated based on the cyclic shifting method.

[0067] Of course, during the training phase, after a set of coordinates corresponding to the absolute order with the smallest distance deviation is output to the next network (ROI Align network), the target detection model will obtain the first oblique frame of the first image.

[0068] A loss function is obtained according to the deviations between the four points on the first oblique frame and the four label points on the first image, and the parameters in the target detection model are reversely updated by the loss function;

[0069] Of course, the way of calculating the deviations between the 4 points on the first oblique frame and the 4 label points on the first image is the same as the way of calculating the deviations described above.

[0070] For example, the coordinates of the center points of the four label points of the first image can be understood as the coordinates of the center point of the minimum circumscribed rectangle in the above embodiment. The four points on the first oblique frame can be understood as four target points.

[0071] In this way, the distance deviations between the four points on the first oblique frame and the rectangular frames corresponding to the four label points are calculated by calculating the distance deviations between the four target points and the minimum circumscribed rectangle.

[0072] Of course, in practical applications, other methods can be used to obtain the target order and loss function.

[0073] As another example of obtaining the target order and loss function, the distance deviations between the four target points corresponding to the multiple groups of absolute orders obtained after the cyclic shift and the rectangular boxes corresponding to the label points can also be calculated, and the absolute order with the smallest distance deviation can be used as the target order; and the loss function can be generated based on the value with the smallest distance deviation.

[0074] The embodiments of the present application do not limit the specific implementation method.

[0075] After reaching the number of training times (for example, after training 1,000 images), or after the object detection model converges, a trained object detection model is obtained.

[0076] It should be noted that after reaching the training times (for example, after training 1000 images), or after the object detection model converges, the cyclic shift method of obtaining the target order of the points input to the next network (ROI Align network) from the initial order of the four target points is usually fixed.

[0077] In the detection stage, when the RPN network processes the image, the 4-point convex hull method can also be used to obtain the initial order of the 4 target points, and the initial order of the 4 target points includes the relative order of the 4 target points.

[0078] As an example, the four target points are point 5, point 6, point 7, and point 8. Of course, each point has coordinates; for example, the coordinates of point 5 are (x5, y5), the coordinates of point 6 are (x6, y6), the coordinates of point 7 are (x7, y7), and the coordinates of point 8 are (x8, y8). The initial order of the four target points is: 5678.

[0079] The target order can be obtained by a cyclic shift method according to the initial order of the four target points in the training phase. The target order can be obtained according to the initial order 5678.

[0080] As an example, if the initial order of the four target points in the training phase is 1234 and the target order is 2341, then the target order obtained from the initial order 5678 is 6785;

[0081] If the initial order of the four target points in the training phase is 1234 and the target order is 4123, then the target order obtained from the initial order 5678 is 8567;

[0082] If the initial order of the four target points in the training phase is 1234 and the target order is 1234, then the target order obtained from the initial order 5678 is 5678.

[0083] The method for obtaining the target sequence provided by the embodiment of the present application can reduce the oscillation problem in the training process and can also improve the accuracy of obtaining the target sequence in the detection stage, thereby improving the detection accuracy.

[0084] In order to have a clearer understanding of the embodiments of the present application, the training process and the detection process will be described in detail through the target detection model.

[0085] See also Figure 5 , which is a flow chart of the training process provided in an embodiment of the present application.

[0086] Step 101: Acquire a first image, where the first image is an image in a training set.

[0087] First, a training sample set is obtained, which includes multiple training images. In each training image, a target frame of the target in the training image is marked, wherein the target frame may be an oblique frame having four vertices, each of which may be referred to as a label point. The training image in the training set may be recorded as the first image.

[0088] Step 102: extracting a first feature map of the first image through a feature extraction network.

[0089] Step 103: Process the first feature map through the RPN network to obtain four first target points with a target order on the first image.

[0090] The process of the RPN network obtaining the four first target points with target order includes:

[0091] Step 1031: Calculate the initial order of the four first target points on the first image using the convex hull method, where the initial order of the four first target points includes the relative order of the four first target points.

[0092] The coordinates of the four first target points in the initial order are marked as first coordinates.

[0093] Step 1032, changing the initial order of the four first target points while keeping the relative order of the four first target points unchanged, to obtain at least three groups of absolute orders of the four first target points, wherein the absolute orders include the initial order;

[0094] Step 1033, calculating the distance deviation between the four first target points and the minimum circumscribed rectangle of the four first target points in the absolute order;

[0095] Step 1034: taking the absolute order of the four first target points with the smallest distance deviation as the target order.

[0096] The steps of obtaining three groups of absolute orders include:

[0097] Step 10321, cyclically moving the four first target points in the initial order forward by one position respectively, to obtain a first order of the four first target points;

[0098] Step 10322, circularly move the four first target points in the initial order backward by one position respectively to obtain a second order of the four first target points, and at least three groups of absolute orders of the four first target points include the first order, the second order and the initial order.

[0099] Step 104: input the first feature map and the third coordinates of the four first target points in the target order on the first image into the ROI Align network and the regression network to obtain the first image including at least one first oblique frame.

[0100] Among them, the third coordinate is the coordinate of the four first target points in the target order.

[0101] Step 105, according to the distance deviation between the second coordinates of the four vertices of the first oblique frame and the label coordinates of the label points calibrated on the first image, reversely update the parameters in the target detection model to obtain a trained target detection model.

[0102] The second coordinates are the coordinates of the first oblique frame output by the target detection model in the output order.

[0103] See also Figure 6 , which is a flow chart of the detection process provided in an embodiment of the present application.

[0104] Step 201, obtaining a second image, where the second image is an image to be detected;

[0105] Step 202, extracting a second feature map of the second image through a feature extraction network;

[0106] Step 203, processing the second feature map through the RPN network to obtain four second target points with an initial order on the first image;

[0107] Step 204: reversely calculate the target order of the four second target points according to the order deviation through the RPN network, and the order deviation is a cyclic shift method corresponding to the absolute order with the smallest distance deviation of the four first target points.

[0108] Step 205: input the second feature map and the fourth coordinates of the four second target points in the target order on the second image into the ROI Align network and the regression network to obtain the second image containing at least one second oblique frame. The second oblique frame contains the target in the second image.

[0109] The fourth coordinates are the coordinates of the four second target points in the target order.

[0110] See also Figure 7 , is a second image before detection by the trained target detection model provided by the embodiment of the present application, see Figure 8 , which is a second image detected by the trained target detection model provided by an embodiment of the present application, wherein the second image includes an oblique frame, and the oblique frame contains the target on the second image.

[0111] See also Fig. 9 , is a second image before detection by the trained target detection model provided by the embodiment of the present application, see Fig.10 , which is a second image detected by the trained target detection model provided by an embodiment of the present application, wherein the second image includes an oblique frame, and the oblique frame contains the target on the second image.

[0112] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0113] The embodiment of the present application also provides a target detection device. For the convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0114] The target detection device can be a software unit, a hardware unit, or a combination of software and hardware built into the camera, or can be integrated into the camera as an independent pendant. Of course, it can also be a software unit, a hardware unit, or a combination of software and hardware built into the terminal, or can be integrated into the terminal device as an independent pendant.

[0115] The target detection device comprises:

[0116] A training unit is used to input a first image into a target detection model, and output a first image including at least one first oblique frame, wherein the first image is an image in a training set, and the RPN network is used to generate four first target points with an initial order on the first image, and obtain a target order of the four target points based on first coordinates of the four first target points and a minimum circumscribed rectangle of the four first target points, and the four first target points with the target order are used to generate a first oblique frame;

[0117] An updating unit, configured to reversely update the parameters in the target detection model according to the distance deviation between the second coordinates of the four vertices of the first oblique frame and the label coordinates of the label points calibrated on the first image, so as to obtain a trained target detection model;

[0118] A detection unit is used to input a second image into a trained target detection model, and output a second image containing at least one second oblique frame, wherein the second image is an image to be detected, and the second oblique frame contains the target on the second image. The RPN network is used to generate four second target points with an initial order on the second image, and based on the initial order and the order deviation of the four second target points, a target order of the four second target points is obtained, wherein the order deviation is the deviation between the target order of the four first target points and the initial order of the four first target points, and the target order of the four second target points with the target order is used to generate the second oblique frame.

[0119] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0120] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0121] Fig.11 is a schematic block diagram of a terminal device provided by an embodiment of the present application. The terminal device may be an independent camera (or camera device) or a terminal device integrated with a camera. Fig.11 As shown, the terminal device 11 of this embodiment includes: one or more processors 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110. When the processor 110 executes the computer program 112, the steps in the above-mentioned various method embodiments are implemented, for example Figure 5 Steps 101 to 105 are shown. Or Figure 6 Steps 201 to 205 shown; or, when the processor 110 executes the computer program 112, the functions of each module / unit in the above-mentioned device embodiment are implemented, such as the functions of the training unit, the updating unit and the detection unit shown in the above-mentioned embodiment.

[0122] Exemplarily, the computer program 112 may be divided into one or more modules / units, which are stored in the memory 111 and executed by the processor 110 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 112 in the terminal device 11. For example, the computer program 112 may be divided into a training unit, an updating unit, and a detection unit. Exemplarily:

[0123] A training unit is used to input a first image into a target detection model, and output a first image including at least one first oblique frame, wherein the first image is an image in a training set, and the RPN network is used to generate four first target points with an initial order on the first image, and obtain a target order of the four target points based on first coordinates of the four first target points and a minimum circumscribed rectangle of the four first target points, and the four first target points with the target order are used to generate a first oblique frame;

[0124] An updating unit, configured to reversely update the parameters in the target detection model according to the distance deviation between the second coordinates of the four vertices of the first oblique frame and the label coordinates of the label points calibrated on the first image, so as to obtain a trained target detection model;

[0125] A detection unit is used to input a second image into a trained target detection model, and output a second image containing at least one second oblique frame, wherein the second image is an image to be detected, and the second oblique frame contains the target on the second image. The RPN network is used to generate four second target points with an initial order on the second image, and based on the initial order and the order deviation of the four second target points, a target order of the four second target points is obtained, wherein the order deviation is the deviation between the target order of the four first target points and the initial order of the four first target points, and the target order of the four second target points with the target order is used to generate the second oblique frame.

[0126] Other functions of the above units will not be described in detail.

[0127] The terminal device includes but is not limited to a processor 110 and a memory 111. Those skilled in the art will appreciate that Fig.11 It is only an example of the terminal device 11 and does not constitute a limitation of the terminal device 11. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 11 may also include an input device, an output device, a network access device, a bus, etc.

[0128] The processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0129] The memory 111 may be an internal storage unit of the terminal device 11, such as a hard disk or memory of the terminal device 11. The memory 111 may also be an external storage device of the terminal device 11, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 11. Further, the memory 111 may also include both an internal storage unit of the terminal device 11 and an external storage device. The memory 111 is used to store the computer program and other programs and data required by the terminal device 11. The memory 111 may also be used to temporarily store data that has been output or is to be output.

[0130] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] In the embodiments provided in the present application, it should be understood that the disclosed terminal equipment, devices and methods can be implemented in other ways. For example, the terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by one or more processors, the steps of each of the above-mentioned method embodiments can be implemented.

[0136] Similarly, as a computer program product, when the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0137] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0138] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: Applied to a target detection model including an RPN network, the method comprises: Input a first image into the target detection model, and output the first image including at least one first oblique frame, wherein the first image is an image in a training set, the RPN network is used to generate four first target points with an initial order on the first image, obtain a target order of the four target points based on first coordinates of the four first target points and a minimum circumscribed rectangle of the four first target points, and the four first target points with the target order are used to generate the first oblique frame; According to the distance deviation between the second coordinates of the four vertices of the first oblique frame and the label coordinates of the label points calibrated on the first image, reversely updating the parameters in the target detection model to obtain a trained target detection model; A second image is input into the trained target detection model, and the second image including at least one second oblique frame is output, wherein the second image is an image to be detected, the second oblique frame includes the target on the second image, and the RPN network is used to generate four second target points with an initial order on the second image, and a target order of the four second target points is obtained based on the initial order and the order deviation of the four second target points, wherein the order deviation is the deviation between the target order of the four first target points and the initial order of the four first target points, and the target order of the four second target points with the target order is used to generate the second oblique frame.

2. The method according to claim 1, characterized in that The target detection model includes: a feature extraction network, an RPN network using an ORN algorithm, a ROI Align network, and a regression network connected in sequence, wherein the input of the ROI Align network also includes the output of the feature extraction network; the step of inputting a first image into the target detection model and outputting the first image including at least one first oblique frame includes: Extracting a first feature map of the first image by the feature extraction network; Processing the first feature map through the RPN network to obtain four first target points with a target order on the first image; The first feature map and the third coordinates of the four first target points with target order on the first image are input into the ROI Align network and the regression network to obtain the first image including at least one first oblique frame.

3. The method according to claim 2, characterized in that The step of processing the first feature map through the RPN network to obtain four first target points having a target order on the first image includes: Calculating an initial order of four first target points on the first image using a convex hull method, wherein the initial order of the four first target points includes a relative order of the four first target points; Under the condition that the relative order of the four first target points remains unchanged, changing the initial order of the four first target points to obtain at least three groups of absolute orders of the four first target points, wherein the absolute order includes the initial order; Calculating the distance deviation between the four first target points and the minimum circumscribed rectangle of the four first target points in the absolute order; The absolute order of the four first target points in which the distance deviation is the smallest is used as the target order.

4. The method according to claim 3, characterized in that The method of changing the initial order of the four first target points while keeping the relative order of the four first target points unchanged to obtain at least three groups of absolute orders of the four first target points comprises: The four first target points in the initial order are respectively cyclically moved forward by one position to obtain a first order of the four first target points; The four first target points in the initial order are respectively circularly moved backward by one position to obtain a second order of the four first target points, and at least three groups of absolute orders of the four first target points include the first order, the second order and the initial order.

5. The method according to claim 3, characterized in that The calculating the distance deviation between the four first target points and the minimum circumscribed rectangle of the four first target points in the absolute order includes: Subtract the projection value of the center point of the minimum circumscribed rectangle on the x-axis from the projection value of the first target point in the absolute order on the x-axis to obtain a first deviation of the first target point; Subtract the projection value of the center point of the minimum circumscribed rectangle on the y-axis from the projection value of the second target point in the absolute order on the y-axis to obtain a second deviation of the second target point; Subtract the projection value of the center point of the minimum circumscribed rectangle on the x-axis from the projection value of the third target point in the absolute order on the x-axis to obtain a third deviation of the third target point; Subtracting the projection value of the center point of the minimum circumscribed rectangle on the y-axis from the projection value of the fourth target point in the absolute order on the y-axis to obtain a fourth deviation of the fourth target point; The distance deviation is obtained by calculating the sum of the absolute value of the first deviation, the absolute value of the second deviation, the absolute value of the third deviation, and the absolute value of the fourth deviation.

6. The method according to claim 4, characterized in that The sequence deviation is a cyclic shift mode corresponding to the absolute sequence in which the distance deviation of the four first target points is the smallest; the inputting the second image into the trained target detection model and outputting the second image containing at least one second oblique frame comprises: Extracting a second feature map of the second image by the feature extraction network; Processing the second feature map through the RPN network to obtain four second target points in an initial order on the first image; Reversely calculating the target order of the four second target points according to the order deviation through the RPN network; The second feature map and the fourth coordinates of the four second target points with target order on the second image are input into the ROI Align network and the regression network to obtain the second image including at least one second oblique frame.

7. The method according to claim 1, characterized in that The feature extraction network includes sequentially connected block layers and N consecutive stage extraction networks, the first stage extraction network includes sequentially connected Linear Embedding layers and Swin Transformer Block modules, and the N-1th to Nth stage extraction networks include sequentially connected Patch Merging layers and at least two Swin Transformer Block modules.

8. The method according to claim 7, characterized in that The Swin Transformer Block module includes two connected subunits, wherein the first subunit includes: an LN layer, a W-MSA layer, an LN layer and an MLP layer connected in sequence; and the second subunit includes an LN layer, a SW-MSA layer, an LN layer and an MLP layer connected in sequence.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the steps of the method according to any one of claims 1 to 8 are implemented.

Images