An image correction angle determination method and device, electronic equipment and storage medium

Abstract

The application discloses an image correction angle determination method and device, electronic equipment and a storage medium. The method comprises the following steps: identifying a target orientation quadrant contained in an image to be detected; determining a deflection range and a deflection angle to which the target orientation belongs when the target orientation specifies a quadrant according to the target orientation quadrant; and determining a correction angle of the image by using the target orientation quadrant, the deflection range and the deflection angle to which the target orientation belongs when the target orientation specifies a quadrant. By predicting the target orientation quadrant first and then predicting the deflection angle when the target orientation specifies a quadrant, the idea from coarse to fine is adopted, the detection accuracy is higher than that of directly classifying the image by 360 degrees, and the application can be applied to angle detection in any scene and has high universality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and particularly relates to an image correction angle determination method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, due to the image acquisition device or human factors, the acquired image usually has a large tilt angle problem, so before analyzing and identifying the image, the tilt angle of the image needs to be detected to correct the image.

[0003] In related technologies, there is an angle detection method based on projection method, which is only applicable to simple OCR (Optical Character Recognition) scene (such as PDF document, ID card, passport, driver's license, bank card, etc. correction processing), and is greatly affected by noise, poor in versatility, and a method of using a deep learning model to classify the input image in 360 degrees, which can only get an approximate angle interval and cannot get an accurate angle.

[0004] Therefore, the angle detection method used in the prior art has the problems of poor versatility and poor effect. SUMMARY

[0005] The present application is to solve the above-mentioned problems of the prior art, and provides an image correction angle determination method, device, electronic equipment and storage medium, which is realized by the following technical scheme.

[0006] The first aspect of the present application provides an image correction angle determination method, which comprises:

[0007] identifying the quadrant of the target orientation contained in the image to be detected;

[0008] determining the deflection range and deflection angle to which the target orientation belongs when the target orientation specifies a quadrant according to the quadrant of the target orientation;

[0009] determining the correction angle of the image by using the quadrant of the target orientation, the deflection range and the deflection angle to which the target orientation belongs when the target orientation specifies a quadrant.

[0010] In some embodiments of the present application, the identification of the quadrant of the target orientation in the image to be detected comprises: inputting the image to be detected into a trained detection model to perform feature extraction on the image by a feature extraction network in the detection model to obtain a first feature map;

[0011] the quadrant prediction network in the detection model acquires the first feature map from the feature extraction network, and predicts and outputs the quadrant of the target orientation in the image according to the first feature map.

[0012] In some embodiments of the present application, the determining of the deflection range and the deflection angle to which the target orientation specified quadrant belongs comprises:

[0013] The angle prediction network in the detection model obtains the first feature map and the quadrant of the target orientation, and determines the deflection range and the deflection angle to which the target orientation specified quadrant belongs according to the first feature map and the quadrant of the target orientation and outputs.

[0014] In some embodiments of the present application, the feature extraction of the image by the feature extraction network in the detection model comprises:

[0015] The image is input to the residual network in the feature extraction network after being convoluted by the convolution layer in the feature extraction network; the image after convolution is processed multiple times by the residual network to obtain the first feature map.

[0016] In some embodiments of the present application, the multiple times of downsampling processing of the image after convolution by the residual network to obtain the first feature map comprises:

[0017] The image after convolution is processed at least three times by the residual network to obtain the first feature map.

[0018] In some embodiments of the present application, the at least three times of downsampling processing of the image after convolution by the residual network to obtain the first feature map comprises:

[0019] The image after convolution is processed by the first residual block in the residual network to obtain the first intermediate feature, and the first intermediate feature is input to the second residual block in the residual network; the second intermediate feature is obtained by processing the first intermediate feature by the second residual block, and the second intermediate feature is input to the third residual block in the residual network; the third intermediate feature is obtained by processing the second intermediate feature by the third residual block, and the third intermediate feature is input to the fourth residual block in the residual network; the first feature map is obtained by processing the third intermediate feature by the fourth residual block.

[0020] In some embodiments of the present application, the quadrant prediction network predicts the quadrant of the target orientation in the image according to the first feature map, comprising:

[0021] The first feature map is input to a quadrant classification network in the quadrant prediction network after being down-sampled by a fifth residual block in the quadrant prediction network; and the quadrant classification network determines a quadrant of the target direction based on the down-sampled first feature map.

[0022] In some embodiments of the present application, the angle prediction network determines a deflection range to which the target direction belongs when the target direction specifies a quadrant based on the first feature map and the quadrant of the target direction, including:

[0023] The first feature map is input to a quadrant classification network in the quadrant prediction network after being down-sampled by a fifth residual block in the quadrant prediction network; and the quadrant classification network determines a quadrant of the target direction based on the down-sampled first feature map.

[0024] In some embodiments of the present application, the angle prediction network determines a deflection range to which the target direction belongs when the target direction specifies a quadrant based on the first feature map and the quadrant of the target direction, including:

[0025] The first feature map is input to a quadrant classification network in the quadrant prediction network after being down-sampled by a fifth residual block in the quadrant prediction network; and the quadrant classification network determines a quadrant of the target direction based on the down-sampled first feature map.

[0026] In some embodiments of the present application, the range prediction network predicts a deflection range to which the target direction belongs when the target direction specifies a quadrant based on the second feature map, including:

[0027] The first feature map is input to a quadrant classification network in the quadrant prediction network after being down-sampled by a fifth residual block in the quadrant prediction network; and the quadrant classification network determines a quadrant of the target direction based on the down-sampled first feature map.

[0028] In some embodiments of the present application, the range prediction network predicts a deflection range to which the target direction belongs when the target direction specifies a quadrant based on the second feature map, including:

[0029] The first feature map is input to a quadrant classification network in the quadrant prediction network after being down-sampled by a fifth residual block in the quadrant prediction network; and the quadrant classification network determines a quadrant of the target direction based on the down-sampled first feature map.

[0030] In some embodiments of the present application, the angle prediction network predicts a deflection angle within a specified deflection range of a specified quadrant of the target orientation based on the third feature map, including:

[0031] The third feature map is input to an angle classification network in the angle prediction network after being down-sampled by a seventh residual block in the angle prediction network; and the angle classification network determines a deflection angle within a specified deflection range of a specified quadrant of the target orientation based on the down-sampled third feature map.

[0032] In some embodiments of the present application, the method further includes a training process of the detection model:

[0033] A sample image is obtained; a quadrant label of a target orientation, a deflection range label and a deflection angle label of the target orientation when the target orientation is specified to a specified quadrant are labeled on the sample image according to a quadrant line and an angle line defined in advance in a rectangular coordinate system; and the labeled sample image is used to train the constructed detection model.

[0034] A second aspect of the present application provides an image correction angle determination device, which includes:

[0035] A quadrant identification module is configured to identify a quadrant of a target orientation contained in an image to be detected;

[0036] A deflection direction determination module is configured to determine a deflection range and a deflection angle of the target orientation when the target orientation is specified to a specified quadrant according to the quadrant of the target orientation;

[0037] A correction angle determination module is configured to determine a correction angle of the image by using the quadrant of the target orientation, the deflection range and the deflection angle of the target orientation when the target orientation is specified to the specified quadrant.

[0038] A third aspect of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to the first aspect when executing the program.

[0039] A fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executable on a processor to implement the steps of the method according to the first aspect.

[0040] Based on the image correction angle determination method and device according to the first aspect and the second aspect, the present application has at least the following advantages:

[0041] The correction angle of the image is determined by performing hierarchical prediction on the image to be detected. Specifically, the quadrant in which the target faces is predicted is first predicted, and then the deflection range and angle of the target facing a specified quadrant are predicted based on that quadrant. This coarse-to-fine prediction approach offers higher angle detection accuracy compared to existing technologies that directly perform 360-degree classification prediction on the image. Furthermore, the technical solution of this invention is applicable to angle detection in any type of scene image, offering advantages such as good correction effect and strong versatility. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0043] Figure 1 This is a flowchart illustrating an embodiment of an image correction angle determination method according to an exemplary embodiment of the present invention;

[0044] Figure 2 According to the present invention Figure 1 The network structure diagram of the detection model shown in the embodiment is illustrated.

[0045] Figure 3A According to the present invention Figure 1 The illustrated embodiment shows a schematic diagram of quadrant allocation in a rectangular coordinate system;

[0046] Figure 3B According to the present invention Figure 3A The illustrated embodiment shows a schematic diagram of the annotation of the sample image;

[0047] Figure 4 According to the present invention Figure 1 The illustrated embodiment presents a schematic diagram of a feature extraction process;

[0048] Figure 5 According to the present invention Figure 4 The embodiment shown illustrates a schematic diagram of the structure of a feature extraction network;

[0049] Figure 6 According to the present invention Figure 1 The illustrated embodiment shows a schematic diagram of the prediction process for the orientation of a target in an image to the quadrant.

[0050] Figure 7 According to the present invention Figure 6 The illustrated embodiment presents a schematic diagram of the structure of a quadrant prediction network;

[0051] Figure 8 According to the present invention Figure 1A schematic diagram of a deflection angle prediction process when a target in an image is oriented to a specified quadrant is shown in the embodiment;

[0052] Figure 9 According to the present application Figure 8 A schematic diagram of the structure of an angle prediction network is shown in the embodiment;

[0053] Figure 10 According to the present application Figure 9 A schematic diagram of the structure of a deflection angle prediction network is shown in the embodiment;

[0054] Figure 11 A schematic diagram of the overall structure of a detection model is shown in the embodiment according to an example embodiment of the present application;

[0055] Figure 12 A schematic diagram of the structure of an image correction angle determination device is shown in the embodiment according to an example embodiment of the present application;

[0056] Figure 13 A schematic diagram of the hardware structure of an electronic device is shown in the embodiment according to an example embodiment of the present application;

[0057] Figure 14 A schematic diagram of the structure of a storage medium is shown in the embodiment according to an example embodiment of the present application. DETAILED DESCRIPTION

[0058] The example embodiments will now be described in detail with reference to the accompanying drawings. If the description of the embodiments with reference to the accompanying drawings is used, identical numbers on different drawings represent the same or similar elements. The embodiments described in the example embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0060] It should be understood that, although the terms first, second, third, etc. can be employed in describing various information used in the application, the information is not to be limited to these terms. These terms are only used to distinguish one category of information from another category of information. For example, a first information can also be termed a second information, and, similarly, a second information can also be termed a first information, without departing from the scope of the application. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "in response to determining."

[0061] To solve the problem of poor universality and poor effect of the angle detection method used in the prior art, the application provides an image correction angle determination method, that is, by identifying the quadrant of the target direction contained in the image to be detected, and determining the deflection range and deflection angle to which the target direction specified quadrant belongs according to the quadrant of the target direction, and then determining the correction angle of the image by using the quadrant of the target direction, the deflection range and deflection angle to which the target direction specified quadrant belongs.

[0062] The technical effects achieved based on the above description are:

[0063] By hierarchical prediction of the image to be detected, the correction angle of the image is determined, that is, the quadrant of the target direction in the image is first predicted, and then the deflection range and deflection angle to which the target direction specified quadrant belongs are predicted according to the quadrant of the target direction. This coarse-to-fine prediction idea has higher angle detection accuracy than the 360-degree classification prediction of the image in the prior art, and the technical solution of the application can be applied to angle detection of any type of scene image (such as ID photo rotation, ID card rotation, etc.), and has the advantages of good correction effect and strong universality.

[0064] In order for those skilled in the art to better understand the application, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0065] Embodiment one:

[0066] Figure 1 An embodiment flowchart of an image correction angle determination method according to an exemplary embodiment of the application is shown, Figure 2 The network structure of the detection model used in the embodiment of the application is shown in combination with Figure 1 and Figure 2 The image correction angle determination method comprises the following steps:

[0067] Step 101: Identify the quadrant of the target direction contained in the image to be detected.

[0068] The quadrant of the target direction contained in the image refers to the approximate area of the target inclination.

[0069] In one possible implementation, a deep learning-based detection model can be used to identify the quadrant of the target orientation in the image, as shown in Figure 2 The image to be detected is input into the trained detection model, the feature extraction network in the detection model is used to extract features of the image to obtain a first feature map, then the quadrant prediction network in the detection model obtains the first feature map from the feature extraction network, and finally the quadrant prediction network predicts the quadrant of the target orientation in the image according to the first feature map and outputs the result.

[0070] The feature extraction network in the detection model belongs to the backbone network of the model, which is used to learn the content in the image and obtain a feature map.

[0071] Before step 101 is performed, the detection model needs to be pre-trained. The model training process includes: first, constructing Figure 2 the network structure as shown in the figure, then obtaining sample images, and labeling the quadrant label of the target orientation in the sample images, the deflection range label and the deflection angle label according to the quadrant lines and angle lines defined in the rectangular coordinate system in advance, and training the constructed detection model using the labeled sample images until the quadrant prediction network and the angle prediction network in the detection model converge.

[0072] As shown in the allocation diagram Figure 3A Two mutually perpendicular and intersecting quadrant lines (dashed lines) define four quadrants, and the lower 1 / 4 area is defined as 0 quadrant, the right 1 / 4 area is defined as 1 quadrant, the upper 1 / 4 area is defined as 2 quadrant, and the left 1 / 4 area is defined as 3 quadrant. Four angle lines drawn from the coordinate origin divide each quadrant. Figure 3A It can be seen that the lower angle line represents 360 degrees / 0 degrees, the right angle line represents 90 degrees, the upper angle line represents 180 degrees, and the left angle line represents 270 degrees.

[0073] As shown in the sample image Figure 3B During the labeling process, the center point between the two eyes is taken as the coordinate origin, the quadrant of the head orientation is labeled as 2 quadrant, then the head is counterclockwise rotated according to the 2 quadrant, so that the head is oriented to the specified quadrant 0 quadrant, and the deflection range when oriented to 0 quadrant is labeled as 315 degrees-360 degrees, then the rotated head is horizontally flipped to flip the head to the deflection range of 0 degrees-45 degrees, and the deflection angle of the head relative to the 0 degree line is labeled.

[0074] The calculation formula of the counterclockwise rotation angle is as follows:

[0075] Rotation angle = (4-angle1) % 4 * 90 (Formula 1)

[0076] In formula 1, angle1 represents the quadrant value of the human head before rotation, that is, the quadrant value of the 0th quadrant is 0, the quadrant value of the 1st quadrant is 1, the quadrant value of the 2nd quadrant is 2, and the quadrant value of the 3rd quadrant is 3. For example, when the quadrant value of the human head is 2, the rotation angle is 180 degrees.

[0077] It should be noted that when the deflection angle is labeled, a line perpendicular to the center line of the two eyes can be drawn first, and then the included angle between the line and the 0th line is measured as the deflection angle.

[0078] Further, if the human head is rotated to the specified quadrant 0th quadrant, and the deflection range is 0-45 degrees, then horizontal flipping is not required, and the deflection angle of the human head relative to the 0th line can be directly measured.

[0079] For the specific implementation process of feature extraction on the image by the feature extraction network, please refer to the description of the following embodiments, which will not be described here.

[0080] For the specific implementation process of predicting the quadrant of the target direction in the image by the quadrant prediction network in the detection model according to the first feature map, please refer to the description of the following embodiments, which will not be described here.

[0081] Step 102: determining the deflection range and deflection angle when the target direction is in the specified quadrant according to the quadrant of the target direction.

[0082] In one possible implementation, as shown in Figure 2 , the first feature map and the quadrant of the target direction are obtained by the angle prediction network in the detection model, and the deflection range and deflection angle when the target direction is in the specified quadrant are determined according to the first feature map and the quadrant of the target direction and output.

[0083] Wherein, the specified quadrant is the quadrant that the target should face after rotation. As shown in Figure 3A , the specified quadrant is the 0th quadrant, and the deflection range when the target direction is in the specified quadrant has two categories, one is 315-360 degrees, and the other is 0-45 degrees, and the deflection angle is the deflection angle relative to the 0th line when the target belongs to the 0-45 degree deflection range.

[0084] For the specific implementation process of determining the deflection range and deflection angle when the target direction is in the specified quadrant by the angle prediction network in the detection model according to the first feature map and the quadrant of the target direction, please refer to the description of the following embodiments, which will not be described here.

[0085] It should be noted that the overall structure of the feature extraction network, the angle prediction network and the quadrant prediction network in the present model is lightweight design, which has the characteristics of fast prediction speed, and is beneficial to application deployment on mobile phones or robots with low processing capacity.

[0086] Step 103: Determine the correction angle of the image by using the quadrant of the target orientation, the deflection range to which the target orientation belongs when the target orientation specifies the quadrant, and the deflection angle.

[0087] Optionally, based on the quadrant distribution mode shown in the above Figure 3A , the calculation formula of the correction angle is as follows:

[0088] Correction angle = angle1*90 + angle2*angle3 (Formula 2)

[0089] In formula 2, angle1 represents the quadrant value corresponding to the quadrant of the target orientation; angle2 represents the value corresponding to the deflection range to which the target orientation belongs when the target orientation specifies the quadrant, in the present embodiment, the value corresponding to the deflection range of 0-45 degrees is 1, and the value corresponding to the deflection range of 315 degrees-360 degrees is-1; angle3 represents the deflection angle relative to the 0 degree line when the target belongs to the 0-45 degree deflection range.

[0090] As shown in the above Figure 3B , the quadrant value of angle1 is 2, the value of angle2 is-1, and the value of angle3 is 30 degrees, which are substituted into the above formula 2 to obtain the correction angle of 150 degrees.

[0091] That is, the target in Figure 3B is rotated clockwise by the correction angle of 150 degrees to obtain the corrected target.

[0092] At this point, the correction angle determination process shown in the above Figure 1 is completed, and the correction angle of the image is determined by hierarchical prediction of the image to be detected, that is, the quadrant of the target orientation in the image is first predicted, and then the deflection range and the deflection angle to which the target orientation belongs when the target orientation specifies the quadrant are predicted according to the quadrant of the target orientation. The prediction idea from coarse to fine has higher angle detection accuracy than the direct 360-degree classification prediction of the image in the prior art, and the technical scheme of the present application can be applied to angle detection of any type of scene image, and has the advantages of good correction effect and strong universality.

[0093] Embodiment two:

[0094] Figure 4 A feature extraction process diagram according to the embodiment shown in the above Figure 1 is shown in the above Figure 5 , which is a structural diagram of the feature extraction network, based on the aboveFigure 1 Based on the embodiments shown above, in combination with Figure 4 and Figure 5 As shown, the feature extraction process includes the following steps:

[0095] Step 201: After the image is convoluted by the convolutional layer in the feature extraction network, it is input to the residual network in the feature extraction network.

[0096] For example, assuming that the input is an image of size 112*112*3, after processing by the convolutional layer with a stride of 1, the output is a feature of size 112*112*32. The resolution of the image feature after convolutional layer processing does not change, but the number of channels increases.

[0097] Step 202: The image after convolutional processing is processed by the residual network multiple times to obtain a first feature map.

[0098] The purpose of multiple downsampling processing is to obtain a first feature map of a predetermined size.

[0099] In one possible implementation, the image after convolutional processing can be downsampled at least three times by the residual network to obtain the first feature map.

[0100] In specific implementation, referring to the residual network structure shown in Figure 5 is composed of four residual blocks connected in series, each residual block is stacked by a series of bottleneck residual layers, and the acquisition process of the first feature map includes: downsampled the image after convolutional processing by the first residual block to obtain a first intermediate feature, and input the first intermediate feature to the second residual block, downsampled the first intermediate feature by the second residual block to obtain a second intermediate feature, and input the second intermediate feature to the third residual block, downsampled the second intermediate feature by the third residual block to obtain a third intermediate feature, and input the third intermediate feature to the fourth residual block, and downsampled the third intermediate feature by the fourth residual block to obtain the first feature map.

[0101] It should be noted that the input image can be subjected to a series of downsampling operations by the four residual blocks, gradually reducing the resolution and increasing the number of channels. Based on the example of step 201 above, the feature of size 112*112*32 is input to the residual network and subjected to 4 times of downsampling processing, and the output is a first feature map of size 14*14*160.

[0102] Since the above four residual blocks are stacked by a series of residual convolutional layers, they all belong to the residual network. By using the residual network for feature extraction, the problems of gradient explosion and gradient disappearance caused by the increase of network depth can be avoided, making the network training easier.

[0103] Those skilled in the art will understand that the number of residual blocks in the residual network can be set according to actual needs, as long as it ensures that the final output feature map is of a certain size. Figure 5 The four residual blocks given are merely illustrative examples.

[0104] This completes the above. Figure 4 The feature extraction process shown uses a residual network to extract content features from the image. This can increase the network depth to extract deeper content features, thereby improving the prediction accuracy of subsequent networks.

[0105] Example 3:

[0106] Figure 6 According to the present invention Figure 1 The illustrated embodiment presents a schematic diagram of the prediction process for the orientation of a target in an image to the quadrant. Figure 7 This is a schematic diagram of the quadrant prediction network structure, based on the above. Figures 1 to 5 Based on the illustrated embodiments, combined with Figure 6 and Figure 7 As shown, the target orientation quadrant prediction process includes the following steps:

[0107] Step 301: After downsampling the first feature map through the fifth residual block in the quadrant prediction network, it is input into the quadrant classification network in the quadrant prediction network.

[0108] The fifth residual block is used to further learn the image content in order to obtain more image information and improve the subsequent classification accuracy.

[0109] Based on the example in step 202 above, a first feature map of size 14*14*160 is output. After downsampling processing by the fifth residual block, a first feature map of size 7*7*1280 is obtained.

[0110] Step 302: Based on the first feature map after downsampling, determine the quadrant in which the target in the image is oriented in the preset rectangular coordinate system using the quadrant classification network.

[0111] Optional, such as Figure 7 The specific structure of the quadrant classification network shown first uses a global average pooling layer to pool the downsampled first feature map to obtain 1*1 multi-channel features. Then, a convolutional layer is used to reduce the dimensionality of the 1*1 multi-channel features, and after activation by the softmax function, 1*1 features with a certain channel dimension are obtained.

[0112] Among them, based on the above Figure 3AThe four quadrant distribution shown, the convolutional layer output is 1*1*4 size feature, each channel 1*1 feature represents the predicted value of one quadrant, through argmax to get the quadrant value angle1 of the target orientation quadrant, the value range of angle1 is [0, 1, 2, 3].

[0113] So far, the above Figure 6 The quadrant prediction process shown can roughly estimate the approximate quadrant area of the target orientation in the image.

[0114] Embodiment four:

[0115] Figure 8 For the present application according to Figure 1 The deflection angle prediction process diagram when the target orientation specifies a quadrant in the image shown in the embodiment, Figure 9 The structure of the angle prediction network, based on the above Figures 1 to 7 The embodiment shown, combined with Figure 8 And Figure 9 The deflection angle prediction process when the target orientation specifies a quadrant includes the following steps:

[0116] Step 401: Rotate the first feature map to obtain the second feature map according to the quadrant of the target orientation through the rotation layer in the angle prediction network, so that the target orientation specifies a quadrant.

[0117] Wherein, the rotation formula of the rotation layer is as follows:

[0118] featuremap_processed=rotate(featuremap,(4-angle1)%4*90) (Formula 3)

[0119] In formula 3, featuremap represents the first feature map, featuremap_processed represents the second feature map obtained after rotation, (4-angle1)%4*90 represents the counterclockwise rotation angle, and angle1 represents the quadrant value of the quadrant of the target orientation output by the above quadrant prediction network.

[0120] Step 402: Obtain the second feature map from the rotation layer through the range prediction network in the angle prediction network, and predict the deflection range to which the target orientation specifies a quadrant based on the second feature map.

[0121] Wherein, after rotating the first feature map, it is equivalent to rotating the target to the specified quadrant 0 quadrant, as Figure 3A Shown in the 0 quadrant region, it is divided into two halves by the 360 degree / 0 degree angle line, one half is 315-360 degrees, and the other half is 0-45 degrees, but the target in the 0 quadrant needs to be further predicted which half range it deflects to.

[0122] In one optional specific embodiment, see Figure 9 The range prediction network structure shown in the figure downsamples the second feature map through the sixth residual block and then inputs it into the range classification network. Based on the downsampled second feature map, the range classification network determines the deflection range to which the target orientation belongs in the specified quadrant.

[0123] The sixth residual block is used to further learn the rotated image content to obtain more image information and improve the subsequent classification accuracy.

[0124] Based on the example in step 202 above, the first feature map of size 14*14*160 is output. After rotation, the feature size is still 14*14*160. After downsampling processing by the sixth residual block, a feature of size 7*7*1280 is obtained.

[0125] Optional, continue as follows Figure 9 The specific structure of the range classification network shown first uses a global average pooling layer to pool the downsampled second feature map to obtain 1*1 multi-channel features. Then, a convolutional layer is used to reduce the dimensionality of the 1*1 multi-channel features, and after activation by the tanh function, a 1*1 feature with a certain channel dimension is obtained.

[0126] Among them, based on the above Figure 3A As shown, the specified quadrant 0 region is divided into two ranges by the 360° / 0° angle line, namely 315°-360° and 0°-45°. Therefore, the convolutional layer outputs a 1*1*2 feature, with each channel's 1*1 feature representing one of the ranges. Then, based on the 1*1*2 feature, the deflection range is determined, and the range prediction value angle2 is output. The value range of angle2 is [-1, 1], where "-1" represents 315°-360° and "1" represents 0°-45°.

[0127] Step 403: Obtain the second feature map from the rotation layer through the deflection angle prediction network in the angle prediction network, obtain the deflection range to which the target is facing the specified quadrant from the range prediction network, and predict the deflection angle based on the second feature map and the deflection range.

[0128] In one optional specific embodiment, such as Figure 10 The deflection angle prediction network structure shown in the diagram involves a flip layer that horizontally flips the second feature map based on the deflection range output by the range prediction network to obtain a third feature map, ensuring that the target's orientation towards a specified quadrant falls within a specified deflection range. Then, the angle prediction network obtains the third feature map from the flip layer and predicts the deflection angle of the target's orientation towards the specified quadrant within the specified deflection range based on the third feature map.

[0129] Wherein, there are two cases for the deflection range output by the range prediction network, one is 315-360 degree deflection range, and the other is 0-45 degree deflection range, in order to facilitate the subsequent angle prediction network to predict the specific deflection angle, it is necessary to transform the target to the specified deflection range 0-45 degrees, so that the flipping layer needs to be judged according to the deflection range, if the deflection range is 315-360 degrees, the horizontal flip is executed, if the deflection range is 0-45 degrees, the horizontal flip is not executed.

[0130] That is to say, when the deflection range output by the range prediction network is 315-360 degrees, the third feature map output by the flipping layer is the feature after horizontal flip, and when the deflection range output by the range prediction network is 0-45 degrees, the third feature map output by the flipping layer is the feature without horizontal flip, that is, the same as the second feature map.

[0131] In an optional embodiment, as shown in the structure of the deflection angle prediction network, Figure 10 for the prediction process of the deflection angle, the third feature map is input to the angle classification network in the angle prediction network after being down-sampled by the seventh residual block in the angle prediction network, and then the angle classification network determines the deflection angle within the specified deflection range of the target towards the specified quadrant based on the third feature map after down-sampling.

[0132] Wherein, the seventh residual block is used to further learn the image content after horizontal flip, so as to obtain more image information and improve the subsequent classification accuracy.

[0133] Optionally, continue as shown in the specific structure of the angle classification network, Figure 10 the third feature map after down-sampling is first processed by the global average pooling layer to obtain 1*1 multi-channel feature, and then the 1*1 multi-channel feature is processed by the convolution layer to reduce the dimension, and after being activated by the softmax function, 1*1 feature with certain channel dimension is obtained.

[0134] Wherein, since the target in the third feature map is towards the specified quadrant and belongs to the specified deflection range 0-45 degrees, in order to improve the angle classification accuracy, the classification can be performed according to the step 1 degree, therefore the convolution layer is designed to output 1*1*46 size feature, and the 1*1 feature of each channel represents one angle, then according to the 1*1*46 feature, the specific deflection angle to which it belongs is judged, and the angle prediction value angle3 is output, the value range of angle3 is [0, 1, 2, 3, 4......45].

[0135] At this point, the above Figure 8The deflection angle prediction process shown rotates the first feature map to a target orientation designated quadrant by rotating layers according to the quadrant predicted by the quadrant prediction network, then predicts the deflection range when the target orientation is in the target orientation designated quadrant by the range prediction network, and then predicts the deflection angle based on the deflection range and the second feature map obtained after rotation by the deflection angle prediction network. The design idea of gradually refining the orientation quadrant-deflection range-deflection angle can well improve the angle prediction accuracy.

[0136] Based on the above description of the embodiments, as Figure 11 To detect the overall structure of the model, an image of 112*112*3 is input, and after feature extraction by the convolutional layers, the first residual block to the fourth residual block in the feature extraction network 10, a first feature map of 14*14*160 is output. The first feature map enters the quadrant prediction network 20, and after a series of processes such as the fifth residual block, the global average pooling layer, the convolutional layer, and the softmax layer in the quadrant prediction network 20, the quadrant angle1 of the target orientation in the image is predicted.

[0137] The first feature map enters the angle prediction network 30, and the rotation layer in the angle prediction network 30 rotates the first feature map to obtain the second feature map, so that the target orientation is designated quadrant. Then, the second feature map is processed by a series of processes such as the sixth residual block, the global average pooling layer, the convolutional layer, and the tanh layer, to predict the deflection range angle2 when the target orientation is in the designated quadrant. Then, the third feature map is obtained by horizontally flipping the second feature map according to the predicted deflection range by the flipping layer in the angle prediction network 30, so that the target orientation is in the designated deflection range. Finally, the third feature map is processed by a series of processes such as the seventh residual block, the global average pooling layer, the convolutional layer, and the softmax layer, to predict the deflection angle angle3 in the designated deflection range of the target orientation.

[0138] Finally, the final correction angle is calculated using angle1, angle2, and angle3 according to formula 2 described in step 104.

[0139] Corresponding to the embodiments of the image correction angle determination method described above, the present application also provides embodiments of an image correction angle determination device.

[0140] Figure 12 The structure of an image correction angle determination device according to an exemplary embodiment of the present application is shown in the structure diagram, which is used to execute the image correction angle determination method provided by any of the above embodiments. As shown in Figure 12 The image correction angle determination device includes:

[0141] The quadrant identification module 510 is configured to identify the quadrant of the target orientation contained in the image to be detected.

[0142] The deflection direction determination module 520 is configured to determine a deflection range and a deflection angle to which the target orientation specified quadrant belongs according to the target orientation specified quadrant.

[0143] The correction angle determination module 530 is configured to determine the correction angle of the image by using the target orientation specified quadrant, the deflection range and the deflection angle to which the target orientation specified quadrant belongs.

[0144] The functions and effects of the above-mentioned device modules are described in detail in the implementation process of the corresponding steps of the above-mentioned method, and will not be repeated here.

[0145] For the device embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The above-mentioned device embodiment is only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the present application. Those skilled in the art can understand and implement it without creative labor.

[0146] The embodiment of the present application also provides an electronic device corresponding to the image correction angle determination method provided by the above-mentioned embodiment, to execute the above-mentioned image correction angle determination method.

[0147] Figure 13 The hardware structure of an electronic device according to an exemplary embodiment of the present application is shown, which includes a communication interface 601, a processor 602, a memory 603 and a bus 604; wherein the communication interface 601, the processor 602 and the memory 603 complete the communication among each other through the bus 604. The processor 602 can execute the image correction angle determination method described above by reading and executing the machine executable instructions corresponding to the control logic of the image correction angle determination method in the memory 603, and the specific content of the method can be referred to the above-mentioned embodiment, which will not be repeated here.

[0148] The memory 603 mentioned in the present application can be any electronic, magnetic, optical or other physical storage device, and can contain stored information such as executable instructions, data, etc. Specifically, the memory 603 can be a RAM (Random Access Memory), a flash memory, a storage drive (such as a hard disk drive), any type of storage disk (such as an optical disk, a DVD, etc.), or a similar storage medium, or a combination thereof. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 601 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0149] The bus 604 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 603 is used to store programs, and the processor 602 executes the programs after receiving execution instructions.

[0150] The processor 602 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits or instructions in the form of software in the processor 602. The processor 602 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-program gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution.

[0151] The electronic device provided by the embodiments of the present application and the image correction angle determination method provided by the embodiments of the present application have the same beneficial effects as the method they adopt, run or implement.

[0152] The present application also provides a computer readable storage medium corresponding to the image correction angle determination method provided by the preceding embodiments. Please refer to Figure 14 As shown in the figure, the computer readable storage medium shown is an optical disk 30, and a computer program (i.e. program product) is stored on the optical disk 30, and when the computer program is run by a processor, the image correction angle determination method provided by any of the preceding embodiments will be executed.

[0153] It should be noted that examples of the computer-readable storage medium can further include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical, magnetic storage mediums, and the like, which are not listed one by one here.

[0154] The computer-readable storage medium provided by the above embodiments of the present application has the same beneficial effects as the method adopted, run or implemented by the application program stored therein, based on the same inventive concept as the image correction angle determination method provided by the embodiments of the present application.

[0155] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0156] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article or apparatus that comprises the recited element.

[0157] The above description is merely illustrative of the application, and not in limitation, and it will be understood by those skilled in the art that any modifications, equivalents, improvements, etc. made within the spirit and principle of the application should be included in the scope of the application.

Claims

1. A method for determining an image correction angle, characterized in that, The method comprises: adopting a detection model to identify a target orientation quadrant contained in an image to be detected; determining a deflection range and a deflection angle to which the target orientation quadrant belongs according to the target orientation quadrant, comprising: rotating a first feature map to obtain a second feature map according to the target orientation quadrant through a rotation layer of an angle prediction network in the detection model, so as to make the target orientation quadrant specify a quadrant; predicting the deflection range to which the target orientation quadrant belongs based on the second feature map, and predicting the deflection angle based on the second feature map and the deflection range; the first feature map is obtained by feature extraction of the image by a feature extraction network in the detection model; determining a correction angle of the image by using the target orientation quadrant, the deflection range and the deflection angle to which the target orientation quadrant belongs when specifying a quadrant.

2. The method of claim 1, wherein, The identification of the target orientation quadrant in the image to be detected comprises: inputting the image to be detected into a trained detection model, so as to perform feature extraction of the image by a feature extraction network in the detection model to obtain a first feature map; acquiring the first feature map by a quadrant prediction network in the detection model, and predicting the target orientation quadrant in the image according to the first feature map and outputting.

3. The method of claim 2, wherein, Before the rotation of the first feature map to obtain the second feature map according to the target orientation quadrant through the rotation layer of the angle prediction network in the detection model, further comprising: acquiring the first feature map and the target orientation quadrant by the angle prediction network in the detection model.

4. The method of claim 2, wherein, The feature extraction of the image by the feature extraction network in the detection model to obtain the first feature map comprises: inputting the image after convolution processing by a convolution layer in the feature extraction network into a residual network in the feature extraction network; performing multiple downsampling processing of the image after convolution processing by the residual network to obtain the first feature map.

5. The method of claim 4, wherein, The multiple downsampling processing of the image after convolution processing by the residual network to obtain the first feature map comprises: performing at least three times of downsampling processing of the image after convolution processing by the residual network to obtain the first feature map.

6. The method of claim 5, wherein, The at least three times of downsampling processing of the image after convolution processing by the residual network to obtain the first feature map comprises: performing downsampling processing of the image after convolution processing by a first residual block in the residual network to obtain a first intermediate feature, and inputting the first intermediate feature into a second residual block in the residual network; performing downsampling processing of the first intermediate feature by the second residual block to obtain a second intermediate feature, and inputting the second intermediate feature into a third residual block in the residual network; performing downsampling processing of the second intermediate feature by the third residual block to obtain a third intermediate feature, and inputting the third intermediate feature into a fourth residual block in the residual network; performing downsampling processing of the third intermediate feature by the fourth residual block to obtain the first feature map.

7. The method of claim 2, wherein, The quadrant prediction network predicts the target orientation quadrant in the image according to the first feature map, comprising: The first feature map is input to a quadrant classification network in the quadrant prediction network after being down-sampled by a fifth residual block in the quadrant prediction network. The quadrant classification network determines the quadrant of the target orientation based on the down-sampled first feature map.

8. The method of claim 3, wherein, Before the second feature map is used to predict the deflection range to which the target orientation specified quadrant belongs, the method further includes: The range prediction network in the angle prediction network obtains the second feature map from the rotation layer.

9. The method of claim 8, wherein, Before the second feature map and the deflection range are used to predict the deflection angle, the method further includes: The deflection angle prediction network in the angle prediction network obtains the second feature map from the rotation layer and the deflection range from the range prediction network.

10. The method of claim 8, wherein, The range prediction network predicts the deflection range to which the target orientation specified quadrant belongs based on the second feature map, and the method further includes: The second feature map is input to a range classification network in the range prediction network after being down-sampled by a sixth residual block in the range prediction network. The range classification network determines the deflection range to which the target orientation specified quadrant belongs based on the down-sampled second feature map.

11. The method of claim 9, wherein, The deflection angle prediction network predicts the deflection angle based on the second feature map and the deflection range, and the method further includes: The deflection angle prediction network in the angle prediction network includes a flip layer that flips the second feature map horizontally based on the deflection range to make the target orientation specified quadrant belong to a specified deflection range. The angle prediction network in the deflection angle prediction network obtains the third feature map from the flip layer and predicts the deflection angle within the specified deflection range of the target orientation specified quadrant based on the third feature map.

12. The method of claim 11, wherein, The angle prediction network predicts the deflection angle within the specified deflection range of the target orientation specified quadrant based on the third feature map, and the method further includes: The third feature map is input to an angle classification network in the angle prediction network after being down-sampled by a seventh residual block in the angle prediction network. The angle classification network determines the deflection angle within the specified deflection range of the target orientation specified quadrant based on the down-sampled third feature map.

13. The method of claim 1, wherein, The method further includes a training process of the detection model: Obtain a sample image; Label the quadrant label of the target orientation, the deflection range label to which the target orientation specified quadrant belongs, and the deflection angle label in the sample image according to the quadrant line and angle line defined in advance in the rectangular coordinate system; Train the constructed detection model using the labeled sample image.

14. An image correction angle determination device, characterized in that, The device includes: A quadrant identification module for identifying the quadrant of the target orientation contained in the image to be detected; A deflection direction determination module for determining the deflection range and deflection angle to which the target orientation specified quadrant belongs based on the quadrant of the target orientation; A correction angle determination module for determining the correction angle of the image using the quadrant of the target orientation, the deflection range to which the target orientation specified quadrant belongs, and the deflection angle. The deflection direction determination module is specifically configured to: rotate, by a rotation layer of an angle prediction network in the detection model, a first feature map according to a quadrant of the target orientation to obtain a second feature map, so that the target orientation is specified to a quadrant; predict a deflection range to which the target orientation specified quadrant belongs based on the second feature map, and predict a deflection angle based on the second feature map and the deflection range; and the first feature map is obtained by performing feature extraction on the image by a feature extraction network in the detection model.

15. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the method according to any one of claims 1-13.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-13.

Images