Methods, apparatus and electronic devices for determining image orientation

Abstract

This application provides a method, apparatus, electronic device, and computer storage medium for determining image orientation. The method includes: detecting connected components of a target image to obtain n blocks; determining m cropping points based on the positions of the n blocks in the target image; cropping the target image according to a preset size using the m cropping points to obtain m cropped images; inputting the m cropped images into a first image orientation classification model to obtain the image orientation corresponding to each of the m cropped images; and determining the target orientation of the target image based on the m image orientations. This application improves the accuracy of determining the target orientation of the entire image by cropping m images according to a preset size using m cropping points, making the cropped images representative.

Description

Technical Field

[0001] This application relates to the field of electronic information technology, and in particular to a method, apparatus, electronic device, and computer storage medium for determining image orientation. Background Technology

[0002] With the rapid development of computer and internet technology, teaching content has become increasingly rich. Nowadays, students have a lot of homework and exams. When encountering difficult problems in their homework or when teachers are grading test papers, users can take pictures of the test papers or homework to convert the paper-based test paper or homework information into images, and then import them into an automatic grading system for relevant processing. This enables the use of automated equipment to grade test paper images or correct homework.

[0003] Automated grading systems include automatic, semi-automatic, and manual online grading systems. Regardless of the system, students or teachers need to upload images of their assignments or exam papers before grading can begin. However, due to the variety of photographic equipment, images vary, resulting in images not always being in the correct orientation. Images rotated 90, 180, or even 270 degrees are very common. Rotated images affect the efficiency and accuracy of automatic and semi-automatic grading systems. Even manual grading systems need to reposition rotated images before grading, impacting both efficiency and user experience.

[0004] Therefore, in this situation, it is necessary to propose a method to determine the image orientation, so that the image uploaded to the grading system can be automatically corrected according to the determined image orientation, which will facilitate subsequent grading based on the corrected image and improve grading efficiency. Summary of the Invention

[0005] In view of this, one of the technical problems solved by the embodiments of this application is to provide a method, apparatus, electronic device and computer storage medium for determining the image orientation, so as to overcome the defect in the prior art that the homework images or test questions uploaded to the grading system have a rotation angle and need to be straightened, thereby affecting the grading efficiency.

[0006] In a first aspect, embodiments of this application provide a method for determining the orientation of an image, the method comprising: detecting connected components of a target image to obtain n blocks, where n is a natural number;

[0007] Based on the positions of the n blocks distributed in the target image, m cropping points are determined, where m is a positive integer;

[0008] Based on the m cropping points, m cropped images are obtained by cropping the target image according to a preset size;

[0009] The m cropped images are input into the first image orientation classification model to obtain the image orientation corresponding to each of the m cropped images. The image orientation is one of 0°, 90°, 180°, 270°, or no orientation.

[0010] The target orientation of the target image is determined based on the m image orientations, wherein the target orientation is one of 0°, 90°, 180°, 270°, or no orientation.

[0011] Secondly, embodiments of this application provide an image orientation determination device, the device including a connected component detection module, a cropping point determination module, a cropping module, an image orientation determination module, and a target orientation determination module;

[0012] The connected component detection module is used to detect connected components in the target image and obtain n blocks, where n is a natural number.

[0013] The cropping point determination module is used to determine m cropping points based on the positions of the n blocks distributed in the target image, where m is a positive integer;

[0014] The cropping module is used to crop m cropped images from the target image according to the m cropping points and their positions in the target image according to preset sizes.

[0015] An image orientation determination module is used to input the m cropped images into a first image orientation classification model to obtain the image orientation corresponding to the m cropped images respectively. The image orientation is one of 0°, 90°, 180°, 270°, or no orientation.

[0016] The target direction determination module is used to determine the target direction of the target image based on the m image directions, wherein the target direction is one of 0°, 90°, 180°, 270°, or no direction.

[0017] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the image orientation determination method described in the first aspect.

[0018] Fourthly, embodiments of this application provide a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the image orientation determination method as described in the first aspect or any embodiment of the first aspect.

[0019] This application provides a method, apparatus, electronic device, and computer storage medium for determining image orientation. The method includes: detecting connected components of a target image to obtain n blocks, where n is a natural number; determining m cropping points, where m is a positive integer, based on the positions of the n blocks in the target image; cropping the target image according to a preset size using the m cropping points to obtain m cropped images; inputting the m cropped images into a first image orientation classification model to obtain the image orientation corresponding to each of the m cropped images, where the image orientation is one of 0°, 90°, 180°, 270°, or no orientation; and determining the target orientation of the target image based on the m image orientations, where the target orientation is one of 0°, 90°, 180°, 270°, or no orientation. In this embodiment, m cropping points are determined based on the positions of n blocks in an image. Then, based on the m cropping points, m cropped images are cropped from the image according to a preset size, making the cropped images representative and thus improving the accuracy of the image orientation corresponding to the cropped images. Finally, the target orientation of the entire image is determined based on the m image orientations, further improving the accuracy of judging the target orientation of the entire image. Attached Figure Description

[0020] The following describes some specific embodiments of the present application in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0021] Figure 1 A flowchart illustrating a method for determining image orientation provided in an embodiment of this application;

[0022] Figure 2 This is an application scenario diagram of an image orientation determination method provided in an embodiment of this application;

[0023] Figure 3 A flowchart illustrating another method for determining image orientation provided in an embodiment of this application;

[0024] Figure 4 A flowchart illustrating another method for determining image orientation provided in an embodiment of this application;

[0025] Figure 5 An image orientation determination device provided in an embodiment of this application;

[0026] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0028] It should be noted that the term "target" in this application is merely a singular concept and is not intended to limit or specifically refer to any one image. For example, "target image" refers to any image. The terms "first" and "second" in this application are merely for distinguishing names and do not represent an order. They should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. For example, "first image direction" and "second image direction" can be the same or different. Similarly, "first image direction classification model," "second image direction classification model," and "third image direction classification model" are merely names to distinguish classification models and can be the same or different. In embodiments of this application, when the image direction is "no direction," it indicates that the image is an invalid text image, meaning that the image does not contain text. It may be a blank image or it may contain other non-text elements, such as horizontal lines, tables, binding lines, dashes, graphics, etc.

[0029] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0030] Example 1

[0031] Embodiment 1 of this application provides a method for determining image orientation. Figure 1 A flowchart illustrating a method for determining image orientation provided in an embodiment of this application is shown below. Figure 1 As shown, the method includes the following steps:

[0032] Step S101: Detect the connected components of the target image to obtain n blocks.

[0033] Where n is a natural number, that is, n is a non-negative integer.

[0034] Here, we will first illustrate the application scenarios of the embodiments of this application with examples, such as... Figure 2 As shown, Figure 2In an application scenario of the image orientation determination method provided in this application embodiment, the work image is divided into nine grid regions. A cropping point is found corresponding to each grid. Cropping is performed with the cropping point as the center and a preset size as the size. Each grid corresponds to one cropped image. For ease of description, these nine grids are represented by numbers 1-9. During cropping, there may be cases where a grid is blank, or where the cropping point corresponding to a grid is close to the edge of the work image. For example... Figure 2 Grid 1 is blank. When determining the cropping point of grid 1, the center of the block closest to the center point of grid 1 is used as the cropping point. The cropping point of grid 1 is located in grid 2. During cropping, the cropped images corresponding to grid 1 and grid 2 partially overlap. The cropping point corresponding to grid 7 is close to the edge of the working image. When cropping according to the preset size, it will crop the part outside the working image. In one possible implementation, such as... Figure 2 As shown, the portion of the cropped image corresponding to grid 7 that extends beyond the working image is padded with zeros. Another possible implementation is to move the cropped image corresponding to grid 7 inwards, that is, towards grid 4, ensuring that the cropped image includes the block corresponding to the cropping point. This embodiment does not limit this approach. The nine cropped images are input into an image orientation classification model to obtain nine image orientations, as shown... Figure 2 As shown, the nine image orientations are 0°, 90°, 90°, 180°, 180°, 180°, 180°, 0°270°, and 180°. The number of images in each orientation (0°, 90°, 180°, 270°, and no orientation) is counted. The orientation at 180° has the most images. Therefore, 180° is taken as the target orientation of the assignment image. In other words, the assignment image is upside down and needs to be rotated 180° to obtain a positive assignment image, ensuring that the assignment image uploaded to the assignment grading system is positive.

[0035] The target image can be automatically captured or received from other devices. The target image can be a static picture or photograph, or a frame from a dynamic video. This application does not limit the source of the target image. The target image in this application can be a test question image, homework image, text image, etc. The target image can include, but is not limited to, printed text, handwritten text, or a mixture of printed and handwritten text. It can consist of one or more text paragraphs, or text that does not appear in paragraph form, such as text in a table or flowchart. The text can include, but is not limited to, pixel values ​​used to express certain meanings (such as pixel values ​​of Chinese characters, English, or other languages) and numbers, etc. This application does not limit this.

[0036] This application embodiment detects connected components of a target image. A connected component is a set of pixels consisting of adjacent pixels with the same pixel value. Multiple connected components can be obtained. For an image containing text, there is at least one block. This application embodiment represents the detected connected components as blocks, and by detecting connected components of the target image, n blocks are obtained.

[0037] Optionally, in one embodiment of this application, the target image is binarized to obtain a binarized image; connected component detection is performed on the binarized image to obtain n blocks.

[0038] When the pixel values ​​in an image have only two possible values, 0 or 255, meaning the image brightness value has only two states: a value of 0 indicates a black pixel, and a value of 255 indicates a white pixel, this embodiment directly detects the connected components of the image to obtain n blocks. When the pixel values ​​in the image are between 0 and 255, before detecting the connected components, this embodiment also uses the Ostu method to binarize the image, setting the grayscale value of the pixels to 0 or 255, so that the entire image presents only black and white. It is understood that the bimodal method, the P-parameter method, the iterative method, etc., can also be used to binarize the image; this embodiment does not limit this. After obtaining the binarized image, connected component detection is performed on the binarized image to obtain n blocks.

[0039] Step S102: Determine m cropping points based on the positions of the n blocks distributed in the target image.

[0040] Where m is a positive integer.

[0041] When an image contains a large amount of text, the detected connected components also result in a large number of blocks. If the direction of each block is determined using an image orientation classification model, and then the image orientation is determined based on that direction, the computational load is extremely high. This embodiment first selects m cropping points, then crops the image based on these points to create a cropped image of a predetermined size. Finally, the orientation of the cropped image is detected using an image orientation classification model, saving time. Two examples are provided to illustrate the image cropping process. The first example involves uniformly selecting m cropping points on the image.

[0042] The second example involves selecting m cropping points based on the positions of n blocks distributed across the image. In one possible approach, the image contains many blocks, but they are concentrated in the upper left corner. The image can be divided into multiple regions, taking a uniform division into four regions as an example. The process involves traversing the blocks in the image, calculating the distance between each block and the center point of the current region, and using the center of the block with the minimum distance as the cropping point for the current region. This yields the cropping points for the four regions, which are also concentrated in the upper left corner. This avoids cropping points selected in the lower half or right side of the image, resulting in a blank image. It is understood that if the image contains only one block, then in this embodiment, the selected m... All cropping points are the center points of the corresponding blocks. If the image contains 0 blocks, then the m cropping points selected in this embodiment are the center points of the corresponding m regions. In another possible implementation, m cropping points are first selected evenly on the image. If the block positions around the cropping point are relatively far from the cropping point, for example, there are no blocks within 300 pixels of the cropping point, but there are many blocks within 400 pixels of the cropping point, then the cropping point can be used as the center point and the preset size of 400 pixels can be used for cropping. This avoids the situation where cropping with the cropping point as the center point and a size of 300 pixels results in a blank image that cannot represent the true image of the cropping point. A blank image represents an image that does not contain text.

[0043] When cropping an image according to the first example, it is easy for the cropped image to not represent the true image of the cropping point. However, in the second example, which is the embodiment of this application, m cropping points are selected in the image based on the positions of n blocks distributed in the image. The resulting m cropping points are representative, thereby improving the accuracy of the image orientation corresponding to the cropped image.

[0044] Step S103: Based on m cropping points, crop m cropped images from the target image according to preset sizes.

[0045] In this embodiment, m cropped images are obtained by cropping the target image according to m cropping points and a preset size. For example, a 300×300 square is cropped with the cropping point as the center. Because the selected cropping point is representative, the image obtained by cropping according to the preset size based on the cropping point is also representative, thereby improving the accuracy of the image orientation corresponding to the cropped image.

[0046] Step S104: Input the m cropped images into the first image orientation classification model to obtain the image orientations corresponding to the m cropped images respectively.

[0047] Each cropped image corresponds to an image orientation, which can be one of the following: 0°, 90°, 180°, 270°, or no orientation.

[0048] Image orientation indicates the angle at which the image needs to be rotated. For example, if the image orientation of the cropped image is 0°, the cropped image is upright and does not need to be rotated; if the image orientation of the cropped image is 180°, the cropped image is reversed and needs to be rotated 180° to make it upright. When the image orientation is undefined, it means the cropped image does not contain text. When the image orientation is undefined, the image can be left unrotated or rotated by 0°, 90°, 180°, or 270°; this application does not limit this. The first image orientation classification model in this application can be a convolutional neural network such as the VGG-16 model, AlexNet model, Inception model, or LeNet-5 model; this application does not limit this either. As long as it can classify the image orientations of m input cropped images to obtain m image orientations, it is within the protection scope of this application.

[0049] Step S105: Determine the target direction of the target image based on the directions of m images.

[0050] The target direction is one of the following: 0°, 90°, 180°, 270°, or no direction.

[0051] It should be noted that the target direction represents the direction in which the target image needs to be rotated. It can be 0°, 90°, 180°, 270°, or no direction. In this embodiment, the image direction with the most occurrences (0°, 90°, 180°, 270°, or no direction) is counted and taken as the final direction of the image, i.e., the target direction. This application crops m cropped images from the image based on the positions of n blocks in the image, making the m cropped images representative. The m cropped images correspond to m image directions. The target direction of the target image is determined based on the m image directions. Compared with calculating the image directions of all blocks, this embodiment only calculates the representative image directions, reducing computation time. Compared with first uniformly selecting m cropping points on the image, cropping with the cropping points as the center and a preset size as the size to obtain m cropped images, this embodiment crops m cropped images from the image based on the positions of n blocks in the image, thereby improving the accuracy of the image direction.

[0052] Example 2

[0053] Optionally, in one embodiment of this application, step 102 may further include steps 102a and 102b.

[0054] Step 102a: Divide the target image into m regions.

[0055] For example, the image is divided into a 3×3 grid, which is 9 regions. Here, m equals 9. It can be understood that the region division can also be 4×4 or 2×2. This application embodiment does not limit the rules for dividing the regions. It can be uniform division, random division, or division based on the location of the blocks. For example, more regions can be divided in areas with concentrated blocks, and fewer regions can be divided in areas with fewer blocks. This application embodiment does not limit this.

[0056] Step 102b: Based on the positions of the n blocks distributed in the target image, determine the cropping points corresponding to the m regions respectively.

[0057] This application uses the example of dividing an image into a nine-square grid to illustrate the concept. If the image is filled with text, then the target image is filled with blocks. It can be understood that if a certain area of ​​the image is filled with text, and other areas have no text (e.g., text is only in the upper left corner of the image), then all n blocks are distributed in the upper left square grid area, and other areas do not contain blocks. This application first divides the image into regions, then selects a cropping point for each region. The cropping point is determined based on the position of the n blocks in the target image. For example, if the upper left square grid area is filled with blocks, and other areas do not contain blocks, all blocks are traversed, and the center of the block closest to the current region's center point is used as the cropping point. One region corresponds to one cropping point, and m regions correspond to m cropping points. These m cropping points are not randomly selected, ensuring they are representative. This makes the image cropped from the cropping points representative, improving the accuracy of obtaining the image orientation from the cropped image.

[0058] Optionally, in one embodiment of this application, step 102b may include: when n > 0, determining the target block of the current region based on the distance values ​​between the n blocks and the center point of the current region, and the center point of the target block being the clipping point of the current region; when n = 0, the center point of the current region being the clipping point of the current region.

[0059] Blocks are randomly distributed in an image. When dividing an image, the following situations may occur: some images have no blocks, some images contain one block, and some images contain multiple blocks. Taking a test question image divided into a nine-square grid as an example, if the test question image does not contain any blocks (n=0), then the center of each of the nine squares is used as the clipping point for that square. If the test question image contains one block (n=1), then the clipping point of all nine squares is the center point of that block. If the test image contains 4 blocks (n=4), these 4 blocks are block A, block B, block C, and block D, then a target block needs to be selected from block A, block B, block C, and block D as the cropping point of the grid area. There are nine grid areas, and each grid area corresponds to one target block, so a total of 9 target blocks are selected. The target block can be any one of block A, block B, block C, and block D. In this embodiment, the target block of the current area is determined based on the position of block A, block B, block C, and block D in the test image and the center point of the current area, and the center point of the target block is used as the cropping point of the current area. If cropping is performed solely based on the center point of the region, the region may contain blocks, but these blocks might be located at the edge of the region. If the cropping size is smaller than the distance between the block and the center point, a blank image might be cropped. A blank image has no orientation, but in reality, the region contains blocks, and the image orientation is not directionless; it could be 0°, 90°, 180°, or 270°. This application's embodiment considers whether the test image contains blocks. When the test image does not contain blocks, the center point of the current region is used as the cropping point. When the test image contains blocks, the target block of the current region is determined based on the distances between n blocks and the center point of the current region. This ensures that the cropping point obtained in this way will always contain a block, avoiding situations where the test image contains blocks, but the selected cropping point does not. This prevents the cropped image obtained from cropping based on the cropping point from being a blank image, improving the accuracy of the image orientation.

[0060] Optionally, in one embodiment of this application, the distances between the n blocks and the center point of the current region are calculated based on the positions of the n blocks in the target image, resulting in n distance values; the block corresponding to the minimum value among the n distance values ​​is taken as the target block of the current region.

[0061] For example, taking a test question image divided into a nine-square grid and determining the target block in the upper left corner region as an example, the test question image includes blocks A, B, C, and D. The distances between blocks A, B, C, and D and the center point of the upper left corner region are calculated, yielding distances A, B, C, and D. If the minimum value among these four distances is distance A, then block A corresponding to distance A is selected as the target block of the upper left corner region. This embodiment of the application uses the block corresponding to the minimum distance value as the target block and the center point of the target block as the cropping point. This ensures that a block exists at the cropping point, avoiding situations where the test question image contains a block, but the selected cropping point does not contain a block. This maximizes the inclusion of blocks in the cropped image, thereby improving the result of cropping image direction determination.

[0062] Optionally, in one embodiment of this application, the image is cropped to a preset size, and the method further includes: when cropping according to the preset size based on the cropping point, if the cropping area is larger than the boundary of the target image, then a preset pixel value is added to the cropping area that exceeds the boundary of the target image.

[0063] When cropping according to a preset size based on a cropping point, the preset size may be larger than the image boundary. For example, if the cropping point is located at the edge of the image, cropping at the preset size will crop pixels outside the image. If the cropping area is larger than the boundary of the target image, one possible approach is to move the cropped image inward to ensure that the cropped image includes the block corresponding to the cropping point; another possible approach is to pad the cropped area beyond the boundary with a preset pixel value. In this embodiment, the preset pixel value is set to 0, which is a zero-padding operation. Zero-padding is performed on the portion of the cropped image that exceeds the boundary of the target image. For example, if the left side exceeds the boundary of the target image by 100 pixels, these 100 pixels are set to the preset pixel value 0. If both the left and right sides exceed the boundary of the target image, zero-padding is performed on both sides. Padding the cropped area beyond the boundary of the target image with a preset pixel value ensures that the size of the cropped image is always the preset size, which is beneficial for recognition by the first image orientation classification model, thereby improving the accuracy of image orientation determination for the cropped image.

[0064] Example 3

[0065] Embodiment 3 of this application provides another method for determining image orientation. Figure 3 A flowchart illustrating a method for determining image orientation provided in an embodiment of this application is shown below. Figure 3 As shown, the method includes the following steps:

[0066] Step 201: Detect the connected components of the target image to obtain n blocks, where n is a natural number.

[0067] Step 201 is the same as step 101, and will not be repeated here.

[0068] Optionally, in one embodiment of this application, step 201 may include: detecting connected components of the target image to obtain k blocks, where k is an integer greater than or equal to n; filtering out blocks from the k blocks that satisfy an aspect ratio greater than a preset ratio and / or an area greater than the average area of ​​the k blocks to obtain n blocks.

[0069] The image includes not only text but also non-text elements such as horizontal lines, binding lines, and tables. This application sets filtering rules. One possible approach is to filter out non-text areas, specifically blocks with an aspect ratio greater than a preset value, such as blocks with an aspect ratio greater than 5 or less than 0.2. Normal text blocks typically have an aspect ratio of around 1. When the aspect ratio of a connected component is greater than 5 or less than 0.2, the connected component may be a horizontal line, binding line, or other type of line. Another possible approach is to filter out blocks with an area greater than the average area of ​​k blocks. When the area of ​​a block is greater than the average area of ​​k blocks, the connected component may be a table. Yet another possible approach is to filter out blocks that satisfy both an aspect ratio greater than a preset value and an area greater than the average area of ​​k blocks. For example, a connected component that simultaneously satisfies an aspect ratio greater than 5 and an area greater than the average area of ​​k blocks may be a narrow, elongated table. Based on the set filtering rules, horizontal lines, binding lines, tables, and other blocks in k blocks are filtered out, while text blocks are retained. This can reduce non-text blocks such as tables or graphics in the image and reduce the impact of non-text blocks on determining the image orientation.

[0070] Step 202: Determine the first image orientation of the target image based on the second image orientation classification model and n blocks.

[0071] The orientation of the first image is one of 0°, 90°, 180°, 270°, or no orientation.

[0072] The second image orientation classification model in this application embodiment can be a convolutional neural network such as the VGG-16 model, AlexNet model, Inception model, or LeNet-5 model; this application embodiment does not impose any limitations on this. The first image orientation in this application embodiment refers to the image orientation of the target image; the first image orientation is used to distinguish it from the second image orientation.

[0073] Optionally, in one embodiment of this application, step 202 may include: inputting n blocks into a second image orientation classification model to obtain n block orientations, one block corresponding to one block orientation, wherein the block orientation is one of 0°, 90°, 180°, 270°, or no orientation; according to the n block orientations, counting the number of blocks in the 0°, 90°, 180°, 270°, and no orientation respectively; and taking the block orientation corresponding to the maximum value among the block orientations as the first image orientation.

[0074] In this embodiment, n blocks are input into the second image orientation classification model to obtain n block orientations. One block corresponds to one block orientation. The block orientation can be 0°, 90°, 180°, 270°, or no orientation. The number of blocks in each orientation is counted, and the block orientation corresponding to the maximum number of blocks is taken as the first image orientation.

[0075] Step 203: Determine m cropping points based on the positions of the n blocks distributed in the target image, where m is a positive integer.

[0076] Step 203 is the same as step 102, and will not be repeated here.

[0077] Step 204: Based on m cropping points, crop m cropped images from the target image according to preset sizes.

[0078] Step 204 is the same as step 103, and will not be repeated here.

[0079] Step 205: Input the m cropped images into the first image orientation classification model to obtain the image orientation corresponding to each of the m cropped images. The image orientation is one of 0°, 90°, 180°, 270°, or no orientation.

[0080] Step 205 is the same as step 104, and will not be repeated here.

[0081] It should be noted that there is no specific order between steps 203-205 and step 202. Step 202 can be executed first, followed by steps 203-205, or steps 203-205 can be executed first, followed by step 202, or steps 202 and 203-205 can be executed simultaneously. This application embodiment does not impose any restrictions on this.

[0082] Step 206: Determine the target direction based on the direction of the first image and the directions of the m images.

[0083] The target direction is one of the following: 0°, 90°, 180°, 270°, or no direction.

[0084] In one possible approach, merging the first image direction with m image directions yields m+1 image directions. By counting the number of image directions at 0°, 90°, 180°, 270°, and no direction, the image direction with the highest count is taken as the final image direction, which is the target direction.

[0085] In another possible approach, since the first image orientation is determined based on n blocks, it contributes significantly to the determination of image orientation and has a large weight. The image orientation corresponding to each cropped image has a small weight. When counting the number of image orientations at 0°, 90°, 180°, 270°, and no orientation, the number of orientations corresponding to the first image orientation is multiplied by a weight value. The weight value can be set according to the actual situation. The weight value is used to balance the accuracy of the image orientation determined by the blocks and the image orientation determined by the cropped images. In this embodiment, weight values ​​are set based on the number of cropped images. Taking m=9 as an example, the image is divided into a 3x3 grid, with one grid corresponding to one cropped image, resulting in 3×3 cropped images. The weight value is set to the number of cropped images divided by 3, which is 3. The weight value corresponding to the first image direction is 3, and the weight values ​​corresponding to the m image directions are all 1; or, the weight value corresponding to the first image direction is 0.3, and the weight values ​​corresponding to the m image directions are all 0.1. This embodiment does not limit the choice, as long as the weight value corresponding to the first image direction is greater than the weight values ​​corresponding to the m image directions. Here's a specific example to illustrate: the first image direction is 0°, and the nine image directions are 0°, 90°, no direction, no direction, 0°, 90°, 90°, no direction, and no direction. Counting the number of image directions at 0°, 90°, 180°, 270°, and no direction, we get the number of images at 0° as 1×3+1+1, at 90° as 1+1+1, and at no direction as 1+1+1+1. When counting the number of images at 0°, a value is multiplied by 3, and 3 is the weight value corresponding to the first image direction. The direction with the most images is taken as the target direction, that is, 0° is the target direction of the image. This image is positive and does not need to be rotated.

[0086] According to Embodiment 3 of this application, the target direction of the target image is determined based on the first image direction and the m image directions corresponding to the m cropped images. Compared with Embodiment 1, which determines the target direction based on the m cropped image directions, Embodiment 3 not only considers the m image directions of the m cropped images, but also considers the contribution of n blocks to the determination of the image direction, thereby improving the accuracy of the image direction determination.

[0087] Example 4

[0088] In Embodiments 1 and 2, the image orientation is determined based on the m image orientations corresponding to the m cropped images. In Embodiment 3, the image orientation further includes a first image orientation, which is determined based on the n block images corresponding to the n blocks. In Embodiment 4 of this application, the image orientation further includes a second image orientation, which is obtained based on the scaled image. The second image orientation determined in Embodiment 4 can be combined with the image orientation in any of the above embodiments to determine the target orientation of the image; this application does not limit this aspect.

[0089] Optionally, in one embodiment of this application, the target image is scaled to obtain a scaled image; the scaled image is then input into a third image orientation classification model to obtain the second image orientation corresponding to the scaled image.

[0090] The third image orientation classification model in this application embodiment can be a convolutional neural network such as the VGG-16 model, AlexNet model, Inception model, or LeNet-5 model; this application embodiment does not impose any limitations on this. The second image orientation in this application embodiment refers to the image orientation of the scaled image; the second image orientation is used to distinguish it from the first image orientation.

[0091] Because there are various types of photography devices, the images captured are also different, not only in terms of angle but also in size. Therefore, it is necessary to scale the target image. The size of the scaled image can be set according to the actual situation so that the size of the scaled image can be recognized by the third image orientation classification model, and the second image orientation corresponding to the scaled image can be output.

[0092] When Embodiment 4 of this application is combined with the m image directions in Embodiment 1, the target direction is determined based on the m image directions and the second image direction.

[0093] Optionally, the second image direction can be merged with the m image directions to obtain m+1 image directions. By counting the number of image directions at 0°, 90°, 180°, 270°, and no direction, the direction with the most image directions is taken as the target direction of the image.

[0094] When Embodiment 4 of this application is combined with the m image directions in Embodiment 1 and the first image direction in Embodiment 3, the target direction is determined based on the first image direction, the m image directions, and the second image direction. Two possible implementation methods are described below.

[0095] In one possible implementation, the first image direction, the second image direction, and m image directions are combined to obtain m+2 image directions. By counting the number of image directions at 0°, 90°, 180°, 270°, and no direction, the direction with the most image directions is taken as the target direction of the image.

[0096] In another possible approach, based on the first image direction, m image directions, and the second image direction, the number of images at 0°, 90°, 180°, 270°, and no direction are counted respectively; the number of images at 0°, 90°, 180°, 270°, and no direction are weighted and summed to obtain five weight indices; the direction corresponding to the maximum value among the five weight indices is taken as the target direction.

[0097] This application embodiment sets weight values ​​to balance the accuracy of the first image direction for block determination and the image direction for cropping image determination. The weight value corresponding to the first image direction is greater than the weight values ​​corresponding to m image directions, and the weight value corresponding to the first image direction is greater than the weight value corresponding to the second image direction. This application embodiment sets the weight values ​​according to the number of cropped images. Taking m equal to 16 as an example, the image is divided into 4×4 grid areas. The weight value corresponding to the first image direction is set to 4, the weight values ​​corresponding to the 16 image directions are all 1, and the weight value corresponding to the second image direction is 1; or, the weight value corresponding to the first image direction is set to 0.4, the weight values ​​corresponding to the 16 image directions are all set to 0.1, and the weight value corresponding to the second image direction is 0.1. This application embodiment does not limit this. In addition, it should be noted that the weight values ​​corresponding to the 16 image directions can also be different. This application uses the example of the weight values ​​corresponding to the 16 image directions being the same, and does not mean that this application is limited to this.

[0098] To illustrate with a concrete example, divide the image into 2×2 grids. Calculate the weighted sum of the counts in each direction. The first image direction is 180°, and the four other directions are 90°, 180°, no direction, and no direction. The second image direction is 90°. Count the number of image directions at 0°, 90°, 180°, 270°, and no direction. The weight for the first image direction is 0.2, and the weights for the four other directions and the second direction are all 0.1. Therefore, the weight index for the 180° direction is 1×0.2 + 1×0.1, the weight index for the 90° direction is 1×0.1 + 1×0.1, the weight index for the no direction is 1×0.1 + 1×0.1, and the weight indices for the 0° and 270° directions are both 0. Take the image direction with the maximum weight index as the target direction, i.e., 180°. Since the image is upside down, it needs to be rotated 180° to be upright.

[0099] In Embodiment 4 of this application, the target direction of the target image is determined based on the first image direction determined by n blocks, the m image directions corresponding to m cropped images, and the second image direction corresponding to the scaled image. Compared with Embodiments 1 and 3, Embodiment 4 not only considers the m image directions of the m cropped images, but also considers the contribution of the n blocks to the determination of the image direction, and also considers the contribution of the scaled image to the determination of the image direction, thereby improving the accuracy of the image direction determination.

[0100] Furthermore, this application provides an exemplary description of how to determine the orientation of an image based on a single image. Optionally, such as... Figure 4 As shown, Figure 4 This application provides a flowchart for detecting image orientation.

[0101] Step 401: Binarize the acquired initial image to obtain a binarized image;

[0102] Step 402: Find the connected components of the binarized image to obtain at least one block;

[0103] Step 403: Process the blocks and filter out blocks whose aspect ratio is greater than a preset ratio and / or whose area is greater than the average area of ​​all blocks.

[0104] Step 404: Input the block obtained by detecting the binarized image in step 402 or the block processed in step 403 into the classification model to classify the block by direction;

[0105] Step 405: Count the number of blocks in each direction;

[0106] Step 406: Take the direction with the most blocks as the image direction;

[0107] Step 407: Crop the image based on the processed blocks to obtain the cropped image;

[0108] Step 408: Input the cropped image into the classification model and perform orientation classification on the cropped image;

[0109] It should be noted that after step 403, there is no specific order in which steps 404-406 or steps 407-408 are executed. They can be executed simultaneously or in any order. This embodiment of the application does not impose any restrictions on this.

[0110] Step 409: Count the number of image directions obtained in step 406 and the number of image directions corresponding to the cropped image obtained in step 408;

[0111] Step 410: Take the direction with the most occurrences as the final direction of the image.

[0112] This application embodiment binarizes an image to obtain a binarized image, detects the connected components of the binarized image to obtain at least one block, processes the block to filter out non-text blocks such as horizontal lines and tables, classifies the block orientation using a classification model, and determines the orientation of the block with the most blocks as the image orientation. Furthermore, the image is cropped based on the blocks, and the image orientation corresponding to the cropped image is determined using the classification model. Finally, based on the image orientation determined by the blocks and the image orientation corresponding to the cropped image, the final target orientation of the image is determined. This approach considers not only the contribution of blocks to determining the image orientation but also the contribution of the cropped image, thus improving the accuracy of image orientation determination.

[0113] Example 5

[0114] This application provides an image orientation determination device, such as... Figure 5 As shown, Figure 5 An image orientation determination device provided in this application embodiment includes a connected component detection module 501, a cropping point determination module 502, a cropping module 503, an image orientation determination module 504, and a target orientation determination module 505.

[0115] The connected component detection module 501 is used to detect connected components in the target image and obtain n blocks, where n is a positive integer;

[0116] The cropping point determination module 502 is used to determine m cropping points based on the positions of n blocks distributed in the target image, where m is a positive integer;

[0117] The cropping module 503 is used to crop m cropped images from the target image according to m cropping points and preset sizes;

[0118] The image orientation determination module 504 is used to input m cropped images into the first image orientation classification model to obtain the image orientation corresponding to each of the m cropped images. Each cropped image corresponds to one image orientation, which is one of 0°, 90°, 180°, 270°, or no orientation.

[0119] The target orientation determination module 505 is used to determine the target orientation of the target image based on m image orientations. The target orientation is one of 0°, 90°, 180°, 270°, or no orientation.

[0120] Optionally, in one embodiment of this application, the cropping point determination module 502 is further configured to divide the target image into m regions; and determine the cropping points corresponding to the m regions respectively based on the positions of the n blocks distributed in the target image.

[0121] Optionally, in one embodiment of this application, the clipping point determination module 502 is further configured to determine the target block of the current region based on the distance values ​​between the n blocks and the center point of the current region when n > 0, and the center point of the target block is the clipping point of the current region; when n = 0, the center point of the current region is the clipping point of the current region.

[0122] Optionally, in one embodiment of this application, the cropping point determination module 502 is further configured to calculate the distance between the n blocks and the center point of the current region based on the position of the n blocks in the target image, and obtain n distance values; and take the block corresponding to the minimum value among the n distance values ​​as the target block of the current region.

[0123] Optionally, in one embodiment of this application, the device 50 further includes a pixel value supplementation module, which is used to supplement preset pixel values ​​in the cropping area that exceeds the boundary of the target image when cropping according to the cropping point and according to the preset size.

[0124] Optionally, in one embodiment of this application, the device 50 further includes a first image orientation determination module, which is used to determine the first image orientation of the target image based on a second image orientation classification model and n blocks, wherein the first image orientation is one of 0°, 90°, 180°, 270°, or no orientation; the target orientation determination module 505 is further used to determine the target orientation based on the first image orientation and m image orientations.

[0125] Optionally, in one embodiment of this application, the device 50 further includes a second image orientation determination module, which is used to scale the target image to obtain a scaled image; input the scaled image into a third image orientation classification model to obtain the second image orientation corresponding to the scaled image; the target orientation determination module 505 is also used to determine the target orientation based on m image orientations and the second image orientation.

[0126] Optionally, in one embodiment of this application, the target direction determination module 505 is further configured to determine the target direction based on the first image direction, m image directions, and the second image direction.

[0127] Optionally, in one embodiment of this application, the target direction determination module 505 is further configured to count the number of images at 0°, 90°, 180°, 270° and no direction according to the first image direction, m image directions and the second image direction; to perform weighted summation on the number of images at 0°, 90°, 180°, 270° and no direction respectively to obtain five weight indices; and to take the direction corresponding to the maximum value among the five weight indices as the target direction.

[0128] Optionally, in one embodiment of this application, the first image orientation determination module is further configured to input n blocks into the second image orientation classification model to obtain n block orientations, one block corresponding to one block orientation, wherein the block orientation is one of 0°, 90°, 180°, 270°, or no orientation; based on the n block orientations, the number of blocks in the 0°, 90°, 180°, 270°, and no orientations are counted respectively; and the block orientation corresponding to the maximum value among the block orientations is taken as the first image orientation.

[0129] Optionally, in one embodiment of this application, the connected component detection module 501 is further configured to detect connected components of the target image to obtain k blocks, where k is an integer greater than or equal to n; and to filter out blocks in the k blocks that satisfy the aspect ratio greater than a preset ratio and / or the area greater than the average area of ​​the k blocks to obtain n blocks.

[0130] Optionally, in one embodiment of this application, the connected component detection module 501 is further configured to binarize the target image to obtain a binarized image; and to perform connected component detection on the binarized image to obtain n blocks.

[0131] Example 6

[0132] Based on any of the image orientation determination methods described in Embodiments 1 to 4 above, this application provides an electronic device. It should be noted that the image orientation determination method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.), and PCs. Figure 6 As shown, Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device. The electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communication bus 608.

[0133] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0134] Communication interface 604 is used for communication with other electronic devices or servers.

[0135] The processor 602 is used to execute the computer program 610, specifically the relevant steps in the above-described method embodiment for determining the image direction.

[0136] Specifically, computer program 610 may include computer program code, which includes computer operation instructions.

[0137] The processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0138] Memory 606 is used to store computer program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0139] Specifically, computer program 610 can be used to cause processor 602 to perform the following operations:

[0140] In an optional implementation, the computer program 610 is further configured to cause the processor 602 to perform each step in the computer program 610. The specific implementation of each step in the computer program 610 can be found in the corresponding steps and units described in the above-described method embodiment for determining the image direction, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0141] Example 7

[0142] Based on the image orientation determination method described in Embodiments 1 to 4 above, this application provides a computer storage medium storing a computer program that, when executed by a processor, implements the method described in Embodiments 1 to 4.

[0143] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0144] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the image orientation determination method described herein is implemented. Furthermore, when a general-purpose computer accesses the code used to implement the image orientation determination method shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the image orientation determination method shown herein.

[0145] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0146] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0147] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0148] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method of determining the orientation of an image, characterized by, The method comprises: detecting a connected domain of a target image to obtain n blocks, n being a natural number; determining m clipping points according to positions of the n blocks distributed in the target image, m being a positive integer; clipping m clipping images in the target image according to a preset size according to the m clipping points; inputting the m clipping images into a first image direction classification model to obtain image directions corresponding to the m clipping images respectively, the image direction being one of 0°, 90°, 180°, 270° and no direction; determining a target direction of the target image according to the m image directions, the target direction being one of 0°, 90°, 180°, 270° and no direction; wherein, the determining m clipping points according to positions of the n blocks distributed in the target image comprises: dividing the target image into m regions; determining clipping points corresponding to the m regions respectively according to positions of the n blocks distributed in the target image; wherein, when n>0, determining a target block of a current region according to distance values between the n blocks and a center point of the current region, the center point of the current region being the clipping point of the current region, and the target block being a block corresponding to a minimum value in the distance values; when n=0, the center point of the current region is the clipping point of the current region.

2. The method of claim 1, wherein, The determining a target block of a current region according to distance values between the n blocks and a center point of the current region comprises: calculating distances between the n blocks and the center point of the current region according to positions of the n blocks in the target image to obtain n distance values; taking a block corresponding to a minimum value in the n distance values as the target block of the current region.

3. The method of claim 1, wherein, The clipping image is an image of the preset size, and the method further comprises: when clipping according to the clipping points and the preset size, if a clipping region is greater than a boundary of the target image, a preset pixel value is filled in the clipping region exceeding the boundary of the target image.

4. The method of claim 1, wherein, The method further comprises: determining a first image direction of the target image according to a second image direction classification model and the n blocks, wherein the first image direction is one of 0°, 90°, 180°, 270° and no direction; the determining a target direction of the target image according to the m image directions comprises: determining the target direction according to the first image direction and the m image directions.

5. The method of claim 1, wherein, The method further comprises: performing scaling processing on the target image to obtain a scaled image; inputting the scaled image into a third image direction classification model to obtain a second image direction corresponding to the scaled image; the determining a target direction of the target image according to the m image directions comprises: determining the target direction according to the m image directions and the second image direction.

6. The method of claim 5, wherein, The method further comprises: determining a first image direction of the target image according to a second image direction classification model and the n blocks, wherein the first image direction is one of 0°, 90°, 180°, 270° and no direction; The target direction is determined according to the m image directions and the second image direction, including: The target direction is determined according to the first image direction, the m image directions and the second image direction.

7. The method according to claim 5 or 6, characterized in that, The target direction is determined according to the first image direction, the m image directions and the second image direction, including: The number of images in 0°, 90°, 180°, 270° and no direction is respectively counted according to the first image direction, the m image directions and the second image direction; The five weight indexes are obtained by weighted sum of the number of images in 0°, 90°, 180°, 270° and no direction respectively; The direction corresponding to the maximum value in the five weight indexes is taken as the target direction.

8. The method according to claim 4 or 6, characterized in that, The first image direction of the target image is determined according to the second image direction classification model and the n blocks, including: The n block directions are obtained by inputting the n blocks into the second image direction classification model, one block corresponding to one block direction, wherein the block direction is one of 0°, 90°, 180°, 270° and no direction; The number of blocks in 0°, 90°, 180°, 270° and no direction is respectively counted according to the n block directions; The block direction corresponding to the maximum value in the block number is taken as the first image direction.

9. The method of claim 1, wherein, The n blocks are obtained by detecting the connected domain of the target image, including: The k blocks are obtained by detecting the connected domain of the target image, k being an integer greater than or equal to n; The blocks satisfying the condition that the aspect ratio is greater than a preset ratio and / or the area is greater than the average area of the k blocks are filtered out from the k blocks, to obtain the n blocks.

10. The method of claim 1, wherein, The n blocks are obtained by detecting the connected domain of the target image, including: The target image is binarized to obtain a binarized image; The n blocks are obtained by detecting the connected domain of the binarized image.

11. An apparatus for determining the direction of an image, characterized in that The device includes a connected domain detection module, a cropping point determination module, a cropping module, an image direction determination module and a target direction determination module; The connected domain detection module is configured to detect the connected domain of a target image to obtain n blocks, n being a natural number; The cropping point determination module is configured to determine m cropping points according to the positions of the n blocks in the target image, m being a positive integer; The cropping module is configured to crop m cropped images from the target image according to the m cropping points and a preset size; The image direction determination module is configured to input the m cropped images into a first image direction classification model to obtain image directions corresponding to the m cropped images respectively, the image direction being one of 0°, 90°, 180°, 270° and no direction; The target direction determination module is configured to determine a target direction of the target image according to the m image directions, the target direction being one of 0°, 90°, 180°, 270° and no direction; The cropping point determination module is further configured to: divide the target image into m regions; According to positions of the n blocks in the target image, determine the m regions respectively corresponding to the cutting points; wherein, When n>0, according to distance values between the n blocks and the center point of the current region, determine the target block of the current region, the center point of the target block is the cutting point of the current region, and the target block is the block corresponding to the minimum value in the distance values; When n=0, the center point of the current region is the cutting point of the current region.

12. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete mutual communication through the communication bus; The memory is used for storing at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the image direction determination method in any one of claims 1-10.

13. A computer storage medium, having stored thereon a computer program, which is executed by a processor to implement the image direction determination method in any one of claims 1-10.

Images