Text position marking method and device

Abstract

The embodiments of the present application provide a method and device for text position annotation. The text position annotation method includes: determining the target area where the object to be annotated is located and the first corner point coordinates corresponding to the target area in the image to be processed, and the object to be annotated contains text information; obtaining the second corner point coordinates of the annotation reference area and the text position information in the annotation reference area; determining the transformation matrix according to the first corner point coordinates and the second corner point coordinates; determining the text position information in the object to be annotated according to the transformation matrix and the text position information in the annotation reference area and annotating it. The technical solution of the embodiments of the present application can greatly improve the efficiency and accuracy of text position annotation, and greatly shorten the time of text position annotation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text position marking method and device. Background Art

[0002] When handling business (such as online loan business, etc.), users are often required to upload photos of relevant certificates (such as real estate certificates, etc.) through the user terminal (such as mobile phone, etc.). After receiving the certificate photos uploaded by the user, the backend server will extract relevant certificate information (such as the owner of the house, the location of the house, the property certificate number, etc.) from the certificate photos, and then perform corresponding business processing based on the extracted certificate information.

[0003] In order to extract the ID information from the ID photo, it is usually necessary to mark the specific location of the ID information on the ID content page. Currently, the text location annotation of the ID training set mainly relies on manual annotation using annotation tools. However, the manual annotation method using annotation tools is inefficient, time-consuming, and has large subjective errors. If a large number of comprehensive training sample sets are needed to train the algorithm, it will take a very long time, which is not conducive to rapid parameter tuning and verification of the algorithm. Summary of the invention

[0004] The embodiments of the present application provide a text position marking method and device, which can improve the text position marking efficiency and text position marking accuracy to at least a certain extent, and greatly shorten the text position marking time.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a text position annotation method is provided, comprising: determining a target area where an object to be annotated is located and first corner point coordinates corresponding to the target area in an image to be processed, the object to be annotated containing text information; obtaining second corner point coordinates of an annotation reference area and text position information in the annotation reference area; determining a transformation matrix based on the first corner point coordinates and the second corner point coordinates; determining text position information in the object to be annotated based on the transformation matrix and the text position information in the annotation reference area and annotating it.

[0007] According to one aspect of an embodiment of the present application, a text position labeling device is provided, comprising: a first determination unit, used to determine a target area where an object to be labeled is located and first corner point coordinates corresponding to the target area in an image to be processed, wherein the object to be labeled contains text information; an acquisition unit, used to acquire second corner point coordinates of a labeling reference area and text position information in the labeling reference area; a second determination unit, used to determine a transformation matrix according to the first corner point coordinates and the second corner point coordinates; and a labeling unit, used to determine and label the text position information in the object to be labeled according to the transformation matrix and the text position information in the labeling reference area.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the second determination unit is used to determine the transformation matrix according to the first corner point coordinates and the second corner point coordinates before further including: a correction object acquisition unit, used to acquire a correction object from the image to be processed, the correction object being located at any direction angle of the object to be labeled; a correction corner point coordinate determination unit, used to determine the correction corner point coordinates according to the correction object, the correction corner point coordinates being the corner point coordinates in the first corner point coordinates that are closest to the correction object; and a correction unit, used to correct the image to be processed according to the correction corner point coordinates so that the direction of the text information in the object to be labeled is consistent with that of the text information in the labeling reference area.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the correction corner point coordinate determination unit is configured to: obtain the center of mass coordinates of the correction object; determine the corner point coordinates closest to the center of mass coordinates according to the distance between the center of mass coordinates of the correction object and the first corner point coordinates; and use the corner point coordinates closest to the correction corner point coordinates.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the correction unit is configured to: determine the rotation angle and rotation direction of the image to be processed according to the coordinates of the correction corner point and the direction angle of the correction object; and rotate the image to be processed according to the rotation angle and the rotation direction.

[0011] In some embodiments of the present application, based on the aforementioned solution, the calibration object acquisition unit includes: a preprocessing subunit, configured to preprocess the image to be processed to acquire the calibration object.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the preprocessing subunit is configured to: convert the image to be processed from the RGB color space to the HSV color space to obtain a processed image; extract an image with a single color from the processed image to obtain a correction object.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the preprocessing subunit is configured to: convert the image to be processed into a grayscale image and perform denoising and binarization processing to obtain a processed image; extract contours from the processed image to determine the correction object.

[0014] In some embodiments of the present application, based on the aforementioned scheme, extracting the contour from the processed image and determining the correction object include: extracting the contour from the processed image according to a contour function; determining the target contour from the extracted contour according to a first preset condition; and obtaining the correction object based on the target contour.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the first determination unit is configured to: convert the image to be processed into a grayscale image and perform denoising and binarization processing to obtain a processed image; extract the contour from the processed image according to the contour function; and determine the target area where the object to be labeled is located from the extracted contour according to a second preset condition.

[0016] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in the above embodiment is implemented.

[0017] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in the above embodiments.

[0018] In the technical solutions provided in some embodiments of the present application, by determining the target area where the object to be annotated is located and the first corner point coordinates corresponding to the target area in the image to be processed, and obtaining the second corner point coordinates of the annotation reference area, the transformation matrix is ​​determined according to the first corner point coordinates and the second corner point coordinates, and the text position information in the object to be annotated is determined and annotated according to the transformation matrix and the text position information in the annotation reference area, thereby realizing the automation of text position information annotation, avoiding the problems of low efficiency and time-consuming manual annotation using annotation tools, greatly improving the text position annotation efficiency and annotation accuracy, and shortening the text position annotation time.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0022] Figure 2 A flowchart of a text position marking method according to an embodiment of the present application is shown;

[0023] Figure 3 A flowchart of a text position marking method according to an embodiment of the present application is shown;

[0024] Figure 4 A flowchart of a text position marking method according to an embodiment of the present application is shown;

[0025] Figure 5 A flowchart of a text position marking method according to an embodiment of the present application is shown;

[0026] Figure 6 A flowchart of a text position marking method according to an embodiment of the present application is shown;

[0027] Figure 7 A detailed flow chart of a text position marking method according to an embodiment of the present application is shown;

[0028] Figure 8 A schematic diagram showing a process of marking text position information according to an embodiment of the present application is shown;

[0029] Fig. 9 A block diagram of a text position marking device according to an embodiment of the present application is shown;

[0030] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0032] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0035] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0036] like Figure 1 As shown, the system architecture may include a terminal 101 (the terminal 101 may be Figure 1 ), a smartphone as shown in FIG. 1 ), a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the terminal 101 and the server 103. The network 102 may include, but is not limited to, a wireless network and a wired network, wherein the wired network includes, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, and a local area network. The wireless network includes, but is not limited to, at least one of the following: Bluetooth, WI-FI, and near field communication (NFC). The user can use the terminal 101 to interact with the server 103 through the network 102 to receive or send messages, etc.

[0037] In specific implementation, the terminal 101 is used to obtain the image to be processed including the object to be annotated, and send the image to be processed to the server 103. The server 103 is used to receive the image to be processed including the object to be annotated obtained by the terminal 101, and mark the position of the text information in the object to be annotated.

[0038] In this embodiment, the terminal 101 may be an electronic device applied to the user side and having functions such as image acquisition and interactive communication. Specifically, the terminal 101 may be, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a smart wearable device, a shopping guide terminal, a TV with a network access function, etc. Alternatively, the terminal 101 may also be software that can be run in the above electronic devices. Specifically, the terminal 101 may be an app application in an electronic device.

[0039] In this embodiment, the server 103 may be a background service server applied to the back-end side of data processing, having functions such as data processing, text information recognition and extraction, etc. The server 103 may be an electronic device having data calculation, storage and network interaction functions; or it may be software running in the electronic device to provide support for data processing, storage and network interaction. In this embodiment, the number of servers 103 is not specifically limited. The server 103 may be one server, or may be several servers, or a server cluster formed by several servers.

[0040] It should be understood that Figure 1 The number of terminals 101, networks 102 and servers 103 in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminals 101, networks 102 and servers 103. For example, the server 103 may be a server cluster composed of multiple servers.

[0041] In one embodiment of the present application, the terminal 101 sends the acquired image to be processed 104 and the annotated reference area 105 to the server 103. After receiving the image to be processed 104 and the annotated reference area 105, the server 103 determines the first corner point coordinates of the target area where the object to be annotated is located in the image to be processed 104, and acquires the second corner point coordinates of the annotated reference area 105 and the text position information in the annotated reference area.

[0042] In one embodiment of the present application, the server 103 determines a transformation matrix according to the first corner point coordinates and the second corner point coordinates, and then uses a rectangular operation to calculate the text position information in the object to be annotated and annotates it.

[0043] In one embodiment of the present application, after the server 103 annotates the text position information of the object to be annotated in the image to be processed 104 , it sends the annotated image to the terminal 101 so that the terminal 101 can view the annotated image.

[0044] It should be noted that the text position marking method provided in the embodiment of the present application can be executed by the server 103, and accordingly, the text position marking device can be set in the server 103. However, in other embodiments of the present application, the terminal 101 can also have similar functions as the server, so as to execute the text position marking scheme provided in the embodiment of the present application.

[0045] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0046] Figure 2 A flowchart of a text position marking method according to an embodiment of the present application is shown. The text position marking method can be executed by a server, which can be Figure 1 The server 103 shown in FIG. 1 may also be executed by a terminal device, for example, Figure 1 The terminal 101 shown in FIG. Figure 2 As shown, the method includes:

[0047] Step S210, determining a target region where an object to be annotated is located and first corner point coordinates corresponding to the target region in the image to be processed, wherein the object to be annotated includes text information;

[0048] Step S220, obtaining the coordinates of the second corner point of the marking reference area and the text position information in the marking reference area;

[0049] Step S230, determining a transformation matrix according to the first corner point coordinates and the second corner point coordinates;

[0050] Step S240: Determine the text position information in the object to be annotated according to the transformation matrix and the text position information in the annotation reference area and perform annotation.

[0051] These steps are described in detail below.

[0052] In step S210, a target region where an object to be annotated is located and coordinates of a first corner point corresponding to the target region are determined in the image to be processed, and the object to be annotated includes text information.

[0053] Specifically, the image to be processed refers to an image that includes an object to be labeled. In other words, the image content of the image to be processed includes the object to be labeled, wherein the object to be labeled refers to an object that requires annotation of text position information in the object. For example, when photographing an ID card, an image containing the ID card can be obtained. The image is the image to be processed, and the ID card in the image is the object to be labeled.

[0054] The image to be processed can be an image stored in the server background, an image uploaded by the terminal, or a local image of the terminal. The terminal can be provided with a camera, and the image to be processed can be collected in real time through the camera. The terminal can also be externally connected to a device with an image acquisition function, and the image to be processed can be collected in real time through the device. In order to achieve more accurate text location information annotation, the terminal can use the clearest image among multiple images to be processed and the image that is most conducive to determining corner points as the image to be processed.

[0055] The corner point coordinates of the target area may have a plurality of corner point coordinates according to the shape of the target area. For example, if the target area is a quadrilateral, there are four corner point coordinates, and if the target area is a pentagon, there are five corner point coordinates.

[0056] It should be noted that the formats of the images to be processed include but are not limited to bitmap (BMP) format, Personal Computer Exchange (PCX) format, Tagged Image File Format (TIFF), Graphics Interchange Format (GIF), Joint Photographic Expert Group (JPEG) format, Tagged Graphics (TGA), Exchangeable Image File Format (EXIF), Portable Network Graphics (PNG), Scalable Vector Graphics (SVG), Drawing Exchange Format (DXF) and Encapsulated Post Script (EPS) format.

[0057] In one embodiment of the present application, the target area where the object to be labeled is located can be extracted based on the object nomination method. Specifically, the object nomination method is to find out in advance the possible location of the object to be labeled in the image to be processed, and by utilizing the texture, edge and color information in the image, it is ensured that a high recall rate (Intersection-over-Union, IoU) can be maintained when selecting fewer windows. Among them, the object nomination method includes but is not limited to region-based convolutional neural networks (Region-based Convolutional Neural Networks, R-CNN), fast region-based convolutional neural networks (Fast Region-based Convolutional Neural Networks, Fast R-CNN) and faster region-based convolutional neural networks (Faster Region-based Convolutional Neural Networks, Faster R-CNN), etc., which are not limited here.

[0058] In one embodiment of the present application, the target area where the object to be annotated is located in the image to be processed can also be determined by the findContours function of OpenCV. In this embodiment, see Figure 3 , specifically including:

[0059] Step S310, converting the image to be processed into a grayscale image and performing denoising and binarization processing to obtain a processed image;

[0060] Step S320, extracting a contour from the processed image according to a contour function;

[0061] Step S330: determining a target area where the object to be labeled is located from the extracted contour according to a second preset condition.

[0062] In step S310, the image to be processed is converted into a grayscale image and subjected to denoising and binarization processing to obtain a processed image.

[0063] Specifically, the image to be processed is generally based on the RGB color map, which is a mixture of the three primary colors R (red), G (green), and B (blue) in proportion, and the value range of the R component, G component, and B component is 0 to 255. The grayscale of the image is to make each pixel in the image satisfy R = G = B, that is, the value of the red variable, the value of the green variable, and the value of the blue variable are equal. This value is called the grayscale value.

[0064] The conversion formula between RGB value and grayscale is as follows:

[0065] Gray=α×R+β×G+γ×B

[0066] Among them, R, G, and B represent the three components of the color map, α, β, and γ are weights, and α+β+γ = 1. The most common grayscale method is to determine the values ​​of α, β, and γ based on the different sensitivities of the human eye to red, green, and blue, and the values ​​are: 0.299, 0.587, and 0.114 respectively.

[0067] After the grayscale image is obtained by conversion, the grayscale image is denoised. In a specific embodiment, the grayscale image is denoised by wavelet transform to obtain a denoised image. Wavelet transform is a powerful tool for digital signal processing and is known as a "mathematical microscope". Wavelet transform is developed on the basis of windowed Fourier transform. It absorbs the function of windowed Fourier transform to realize the time-frequency localization analysis of the signal, and at the same time has the ability to adaptively adjust the window size, so that the signal can be better observed. The reason why wavelet transform can be used for image processing is that from a mathematical point of view, signal and image processing can be uniformly regarded as signal processing, that is, the image can be understood as a two-dimensional signal.

[0068] The binarization of an image is to convert a grayscale image into a black-and-white binary image that can reflect the overall structure and local features of the image by selecting an appropriate threshold. In short, binarization is to make the grayscale value of each pixel in the pixel matrix of the image 0 (black) or 255 (white), that is, to make the entire image appear only black and white. The grayscale value range in the grayscale image is 0 to 255, and the grayscale value range in the binarized image is 0 or 255.

[0069] After the image to be processed is converted into a grayscale image and subjected to denoising and binarization processing, a processed image is obtained.

[0070] Continue to see Figure 3 In step S320, a contour is extracted from the processed image according to a contour function.

[0071] Specifically, the contour function is the findContours function in the OpenCV image processing library. As an open source software library widely used in the field of image processing, OpenCV provides a large number of basic image processing methods that can be directly used, such as denoising, transformation, binarization, etc. When extracting contours from an image, the findContours function is used to input image parameters in the specified format to extract contours.

[0072] Continue to see Figure 3 In step S330, the target area where the object to be labeled is located is determined from the extracted contour according to the second preset condition.

[0073] Because when calling OpenCV's findContours function, the number and size of the returned contours will be affected by the specific ambient lighting and cluttered background in the image to be processed, so auxiliary judgment of the returned contours is also required.

[0074] The second preset condition can be set according to the length and width information of the image to be processed and the length and width size information of the target area. For example, the length and width of the extracted target area cannot be less than 1 / 10 of the length and width of the image to be processed, the area of ​​the extracted target area cannot be less than 1 / 100 of the area of ​​the image to be processed, and cannot be greater than 2 / 3 of the area of ​​the image to be processed, etc. This application does not make specific limitations here.

[0075] Continue to see Figure 2 In step S220, the coordinates of the second corner point of the marked reference area and the text position information in the marked reference area are obtained.

[0076] The annotation reference area is an area determined by a reference object having the same text information as the object to be annotated. In one embodiment of the present application, the position annotation of the text information in the reference object in the annotation reference area is completed by manual intervention in the early stage. Therefore, in the manual annotation process, the coordinates of the second corner point of the annotation reference area and the text position information in the annotation reference area can be obtained according to measurement and calculation.

[0077] Continue to see Figure 2 In step S230, a transformation matrix is ​​determined according to the first corner point coordinates and the second corner point coordinates.

[0078] In one embodiment of the present application, the transformation matrix determined according to the coordinates of the first corner point and the second corner point can be a perspective transformation matrix obtained by a perspective transformation method. Perspective transformation refers to the condition that the perspective center, the image point, and the target point are collinear, and the image receiving surface (perspective surface) is rotated around the trace (perspective axis) by a certain angle according to the perspective rotation law, thereby destroying the original projection light beam, and still maintaining the projection geometry on the image receiving surface unchanged. The perspective transformation matrix refers to the projection of an image onto a new viewing plane (Viewing Plane) in a perspective transformation, which is also called projective mapping.

[0079] In one embodiment of the present application, the direction of the object to be annotated in the image to be processed may be left or right or reversed or in other directions. Therefore, before determining the transformation matrix according to the first corner point coordinates and the second corner point coordinates, the image to be processed needs to be corrected so that the direction of the text information in the object to be annotated is consistent with the direction of the text information in the annotation reference area. In this embodiment, see Figure 4 , specifically including steps S410 to S430, which are described in detail below:

[0080] In step S410, a calibration object is obtained from the image to be processed, where the calibration object is located at any direction angle of the object to be labeled.

[0081] The correction object is located in the image to be processed and is used to correct the image to be processed. The correction object can be a circle, a pentagon, a hexagon, etc. located in the image to be processed, and the embodiment of the present application is not limited here. The obtained correction object must be located at a certain direction angle of the object to be annotated, and the direction angle of the object to be annotated can be the upper left, upper right, lower left or lower right.

[0082] In one embodiment of the present application, obtaining the calibration object from the image to be processed specifically includes: preprocessing the image to be processed to obtain the calibration object.

[0083] In this embodiment, by preprocessing the image to be processed and then obtaining the correction object based on the processed image, different correction objects can be obtained according to different processing methods of the image to be processed. Two specific processing methods are exemplarily provided in the technical solution of the present application, but do not constitute a limitation on the embodiments of the present application.

[0084] In a specific embodiment, see Figure 5 , preprocessing the image to be processed to obtain the correction object specifically includes:

[0085] Step S510, converting the image to be processed from the RGB color space to the HSV color space to obtain a processed image;

[0086] Step S520: extracting an image having a single color from the processed image to obtain a correction object.

[0087] In step S510, the image to be processed is converted from the RGB color space to the HSV color space to obtain a processed image.

[0088] RGB color space is obtained by changing the three color channels of red (Red, R), green (Green, G) and blue (Blue, B) and superimposing them on each other to obtain various colors. RGB color space includes almost all colors that can be perceived by human vision, and is also one of the most widely used color systems. The images that people usually see or take with terminals are generally RGB images. The RGB color mode uses the RGB model to assign an intensity value in the range of 0 to 255 to the RGB components of each pixel in the image. For example: the R value of pure blue is 0, the G value is 0, and the B value is 255; the three RGB values ​​of gray are equal (except 0 and 255), the R, G, and B values ​​of white are all 255; the R, G, and B values ​​of black are all 0. RGB images only use three colors to mix them in different proportions, and 16777216 colors appear on the screen.

[0089] The HSV color space is a color space created based on the intuitive characteristics of color. The color parameters in this space are hue (H), saturation (S), and value (V). Among them, H is measured by angle, with a value range of 0°~360°. Starting from red and counting in a counterclockwise direction, red is 0°, green is 120°, and blue is 240°. Their complementary colors are: yellow is 60°, cyan is 180°, and magenta is 300°; S represents the degree to which the color is close to the spectral color. A color can be regarded as the result of mixing a certain spectral color with white. The larger the proportion of the spectral color, the higher the degree to which the color is close to the spectral color, and the higher the saturation of the color. The higher the saturation, the deeper and brighter the color. The white light component of the spectral color is 0, and the saturation reaches the highest. The value range is usually 0%~100%. The larger the value, the more saturated the color. Lightness V indicates the brightness of a color. For light source color, the lightness value is related to the brightness of the light source; for object color, this value is related to the transmittance or reflectance of the object. The value range is usually 0% (black) to 100% (white).

[0090] For an RGB image, it can be converted to an HSV image, and for an HSV image, it can also be converted to an RGB image. The RGB color image can be converted from RGB space to HSV space by the following conversion formula:

[0091] V=Max

[0092] Max=max(R,G,B),Min=min(R,G,B)

[0093] Among them, R, G, and B represent color channels.

[0094] The conversion relationship from HSV color space to RGB space is an inverse operation.

[0095] Continue to see Figure 5 In step S520, an image having a single color is extracted from the processed image to obtain a correction object.

[0096] After the image to be processed is converted into the HSV color space to obtain the processed image, an image with a single color can be extracted from the processed image, and the extracted single color image is used as a correction object. In a specific embodiment, the single color image can be a green circle.

[0097] In another embodiment, see Figure 6 , preprocessing the image to be processed to obtain the correction object specifically includes:

[0098] Step S610, converting the image to be processed into a grayscale image and performing denoising and binarization processing to obtain a processed image;

[0099] Step S620: extracting contours from the processed image to determine a correction object.

[0100] Step S610 is similar to step S310 and will not be described again here.

[0101] In step S620, a contour is extracted from the processed image to determine a correction object.

[0102] In one embodiment of the present application, contours may be extracted from the processed image using the contour function findContours in the OpenCV image processing library, and the correction object may be determined from the extracted contours.

[0103] In one embodiment of the present application, extracting a contour in the processed image and determining a correction object includes:

[0104] extracting contours from the processed image according to a contour function;

[0105] determining a target contour from the extracted contours according to a first preset condition;

[0106] A correction object is obtained based on the target contour.

[0107] In this embodiment, when calling the findContours function of OpenCV, the number and size of the returned contours will be affected by the specific ambient lighting and cluttered background in the image to be processed, so it is necessary to perform auxiliary judgment on the returned contours. The first preset condition can be set according to the actual situation, and this application does not limit it here. The correction object can be determined based on the target contour.

[0108] Continue to see Figure 4In step S420, the coordinates of the correction corner points are determined according to the correction object, and the coordinates of the correction corner points are the coordinates of the corner points that are closest to the correction object in the first corner point coordinates.

[0109] Specifically, after determining the calibration object, the calibration corner point coordinates are determined according to the calibration object, the determined calibration corner point coordinates are one of the first corner point coordinates, and the calibration corner point coordinates are the corner point coordinates closest to the calibration object.

[0110] In one embodiment of the present application, determining the coordinates of the correction corner points according to the correction object includes:

[0111] Obtaining the centroid coordinates of the calibration object;

[0112] Determine the coordinates of the corner point closest to the centroid coordinates according to the distance between the centroid coordinates of the calibration object and the first corner point coordinates;

[0113] The coordinates of the corner point with the closest distance are used as the coordinates of the correction corner point.

[0114] In specific implementation, different methods may be used to obtain the centroid coordinates of the calibration object according to different calibration objects. If the calibration object is determined to be circular, the Hough circle transform detection method may be used.

[0115] The Hough circle transform is a special case of the Hough transform, which is mainly used to detect circular targets. The Hough circle transform uses some boundary points of the circular target to find the coordinates of the center of the circle, thereby restoring the entire circular boundary. The basic idea of ​​the Hough circle transform is to assume that each non-zero pixel on the image may be a potential point on a circle. By voting, a cumulative coordinate plane is generated, and a cumulative weight is set to locate the circle. The standard Hough circle transform combines the Cartesian coordinate system and the three-dimensional coordinate system. Based on the principle that all circles passing through a certain point in the Cartesian coordinate system are mapped to a three-dimensional curve in the three-dimensional coordinate system, it determines whether the circle in the two-dimensional coordinate system corresponding to the three-dimensional point is retained as the final circle fitting result by judging whether the number of intersections of each point in the three-dimensional coordinate system is greater than a certain threshold. To improve computational efficiency, the improved Hough circle transform is directly processed in a two-dimensional coordinate system. For example, all boundary points are regarded as the center of the circle, and a circle is drawn with a radius within the minimum radius and maximum radius in the input parameters. All the obtained circles will have many intersections. The number of intersections at the corresponding pixel points in the image is recorded, and the pixel points greater than the minimum required number of points in the input parameters are recorded as the center points of the fitting circle. The radius of the circle of the same size that intersects the center point the most is used as the radius of the fitting circle, thereby fitting a circular target. Since the determined correction object is circular in shape, the Hough circle transform is used to find the centroid coordinates of the correction object.

[0116] If the determined calibration object is non-circular, for example, a pentagon, the centroid coordinates of the calibration object can be obtained by calling the OpenCV function.

[0117] After determining the centroid coordinates of the calibration object, distance calculation is performed based on the centroid coordinates and the corner point coordinates included in the first corner point coordinates to determine the corner point coordinates closest to the centroid coordinates, and the corner point coordinates closest to the centroid coordinates are used as the calibration corner point coordinates.

[0118] Continue to see Figure 4 In step S430, the image to be processed is corrected according to the correction corner point coordinates so that the direction of the text information in the object to be annotated is consistent with the direction of the text information in the annotation reference area.

[0119] Specifically, the purpose of correcting the processed image is to make the direction of the text information in the object to be labeled consistent with that of the text information in the annotation reference area, wherein the consistent direction means that the direction of the text in the text information is consistent, and the text information of the object to be labeled and the text in the annotation reference area may be directed upward at the same time, or downward at the same time, or left at the same time, or right at the same time. The present application does not make any specific limitation here.

[0120] In one embodiment of the present application, correcting the image to be processed according to the correction corner point coordinates includes:

[0121] Determining the rotation angle and rotation direction of the image to be processed according to the coordinates of the correction corner point and the direction angle of the correction object;

[0122] The image to be processed is rotated according to the rotation angle and the rotation direction.

[0123] In this embodiment, the rotation angle and rotation direction of the image to be processed are determined according to the coordinates of the correction corner point and the direction angle of the correction object. The rotation direction can be counterclockwise or clockwise, and the rotation angle can be any angle.

[0124] For example, if the direction angle of the correction object is that the correction object is located at the upper left corner of the object to be labeled, and the correction corner point coordinates are the corner point coordinates located at the upper right corner of the object to be labeled, wherein, how to determine the direction angle of the correction corner point coordinates is the method of corner point coordinates at which direction angle can be to extract the target area where the object to be labeled is located through the OpenCV function and obtain the first corner point coordinates of the target area, and arrange the first corner point coordinates of the target area according to the upper left, upper right, lower right, and lower left positions through the function input setting. Therefore, after determining the correction corner point coordinates, it can be determined that the correction corner point coordinates are the corner point coordinates at a certain direction angle of the object to be labeled.

[0125] Continuing with the above example, if the direction angle of the correction object is that the correction object is located at the upper left corner of the object to be labeled, and the correction corner point coordinates are the corner point coordinates located at the upper right corner of the object to be labeled, then according to the correction corner point coordinates and the direction angle of the correction object, it can be determined that the rotation direction is clockwise and the rotation angle is 90 degrees. Therefore, the image to be processed can be rotated 90 degrees clockwise to straighten the object to be labeled.

[0126] Continue to see Figure 2 In step S240, the text position information in the object to be annotated is determined and annotated according to the transformation matrix and the text position information in the annotation reference area.

[0127] After the transformation matrix is ​​obtained in step S230, the text position information in the object to be annotated can be determined based on the transformation matrix and the text position information in the annotation reference area. After the text position information is obtained, the corresponding text position information can be annotated to obtain the annotated image.

[0128] Specifically, the following transformation formula can be used to obtain the text position information in the object to be annotated:

[0129]

[0130] in, is the transformation matrix, [u, v, w] is the coordinate before the transformation, that is, the text position information in the annotated reference area, and the coordinate after the transformation [x, y] is obtained according to the coordinate before the transformation, that is, the text position information in the object to be annotated. Since the two-dimensional image is processed, the coordinate w before the transformation is always 1. The coordinate after the transformation [x, y] can be calculated by the following two formulas:

[0131]

[0132]

[0133] It should be noted that if the transformed coordinates need to be converted into two dimensions, a 33 is always 1, so the a in the above two formulas can be 33 The value is 1.

[0134] Through the technical solution in the above embodiment, the target area where the object to be annotated is located and the first corner point coordinates corresponding to the target area are determined in the image to be processed, and the second corner point coordinates of the annotation reference area are obtained, and the transformation matrix is ​​determined according to the first corner point coordinates and the second corner point coordinates. According to the transformation matrix and the text position information in the annotation reference area, the text position information in the object to be annotated is determined and annotated, thereby realizing the automation of text position information annotation, avoiding the low efficiency and time-consuming problems caused by manual annotation using annotation tools, greatly improving the text position annotation efficiency and annotation accuracy, and shortening the text position annotation time.

[0135] Figure 7 A detailed flow chart of a text position marking method according to an embodiment of the present application is shown. Figure 7 As shown, the specific steps include:

[0136] Step S710: determining the coordinates of a first corner point corresponding to a target area where an object to be annotated is located in the image to be processed.

[0137] In a specific embodiment, the image needs to be preprocessed before determining the coordinates of the first corner point corresponding to the target area. The image preprocessing method can be to convert the image to be processed into a grayscale image and perform denoising and binarization to obtain a processed image; extract the contour from the processed image according to the contour function; determine the target area where the object to be marked is located and the coordinates of the first corner point corresponding to the target area from the extracted contour.

[0138] Step S720: Obtain the coordinates of the second corner point of the annotation reference area and the text position information in the annotation reference area.

[0139] The annotation reference area is an area determined by a reference object having the same text information as the object to be annotated. In one embodiment of the present application, the position annotation of the text information in the reference object in the annotation reference area is completed by manual intervention in the early stage. Therefore, in the manual annotation process, the coordinates of the second corner point of the annotation reference area and the text position information in the annotation reference area can be obtained according to measurement and calculation.

[0140] Step S730: Calculate and obtain transformation evidence.

[0141] Specifically, after obtaining the first corner point coordinates in step S710 and obtaining the second corner point coordinates in step S720, the transformation matrix may be obtained by a perspective transformation matrix method.

[0142] Step S740: Determine text position information in the object to be annotated.

[0143] The text position information in the object to be annotated is determined by matrix operation based on the text position information in the annotation reference area and the transformation matrix.

[0144] Figure 8 A schematic diagram of a text position information annotation process according to an embodiment of the present application is shown.

[0145] First, the coordinates of the second corner point and the text position information in the marking reference area 901 are obtained. The text position information has been marked in the marking reference area, that is, the text information has been marked by a rectangular frame.

[0146] Secondly, the target area of ​​the object to be annotated and the coordinates of the first corner point of the target area object are determined in the processed image 902. The object to be annotated contains text information, and the text information in the object to be annotated is consistent with the text information contained in the annotation reference area in terms of text content and text position.

[0147] Again, a transformation matrix is ​​obtained based on the determined coordinates of the first corner point and the second corner point, and then a labeled image 903 can be obtained through matrix operations based on the transformation matrix and the text position information in the labeled reference area 901. In the labeled image 903, the text information is labeled in the form of a rectangular frame.

[0148] The following describes an apparatus embodiment of the present application, which can be used to execute the text position marking method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the text position marking method of the present application.

[0149] Fig. 9 A block diagram of a text position marking device according to an embodiment of the present application is shown, referring to Fig. 9 As shown, a text position tagging device 900 according to an embodiment of the present application includes: a first determining unit 902 , an acquiring unit 904 , a second determining unit 906 and a tagging unit 908 .

[0150] Among them, the first determination unit 902 is used to determine the target area where the object to be annotated is located and the first corner point coordinates corresponding to the target area in the image to be processed, and the object to be annotated contains text information; the acquisition unit 904 is used to obtain the second corner point coordinates of the annotation reference area and the text position information in the annotation reference area; the second determination unit 906 is used to determine the transformation matrix according to the first corner point coordinates and the second corner point coordinates; the annotation unit 908 is used to determine the text position information in the object to be annotated according to the transformation matrix and the text position information in the annotation reference area and perform annotation.

[0151] In some embodiments of the present application, the second determination unit 906 is used to determine the transformation matrix according to the first corner point coordinates and the second corner point coordinates before further including: a correction object acquisition unit, used to acquire a correction object from the image to be processed, the correction object being located at any direction angle of the object to be labeled; a correction corner point coordinate determination unit, used to determine the correction corner point coordinates according to the correction object, the correction corner point coordinates being the corner point coordinates in the first corner point coordinates that are closest to the correction object; and a correction unit, used to correct the image to be processed according to the correction corner point coordinates so that the direction of the text information in the object to be labeled is consistent with that of the text information in the labeling reference area.

[0152] In some embodiments of the present application, the correction corner point coordinate determination unit is configured to: obtain the center of mass coordinates of the correction object; determine the corner point coordinates closest to the center of mass coordinates based on the distance between the center of mass coordinates of the correction object and the first corner point coordinates; and use the corner point coordinates closest to the correction corner point coordinates.

[0153] In some embodiments of the present application, the correction unit is configured to: determine the rotation angle and rotation direction of the image to be processed based on the coordinates of the correction corner point and the direction angle of the correction object; and rotate the image to be processed according to the rotation angle and the rotation direction.

[0154] In some embodiments of the present application, the calibration object acquisition unit includes: a preprocessing subunit, configured to preprocess the image to be processed to acquire the calibration object.

[0155] In some embodiments of the present application, the preprocessing subunit is configured to: convert the image to be processed from the RGB color space to the HSV color space to obtain a processed image; extract an image with a single color from the processed image to obtain a correction object.

[0156] In some embodiments of the present application, the preprocessing subunit is configured to: convert the image to be processed into a grayscale image and perform denoising and binarization processing to obtain a processed image; extract contours from the processed image to determine a correction object.

[0157] In some embodiments of the present application, extracting contours from the processed image and determining a correction object include: extracting contours from the processed image according to a contour function; determining a target contour from the extracted contours according to a first preset condition; and obtaining a correction object based on the target contour.

[0158] In some embodiments of the present application, the first determination unit 902 is configured to: convert the image to be processed into a grayscale image and perform denoising and binarization processing to obtain a processed image; extract a contour from the processed image according to a contour function; and determine a target area where the object to be labeled is located from the extracted contour according to a second preset condition.

[0159] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0160] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0161] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003, such as executing the method described in the above embodiment. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through the bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0162] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0163] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0164] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0165] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0167] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0168] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0169] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0170] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0171] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A text position marking method, characterized in that: include: Determine, in the image to be processed, a target area where an object to be annotated is located and coordinates of a first corner point corresponding to the target area, wherein the object to be annotated includes text information; Acquire the coordinates of the second corner point of the annotation reference area and the text position information in the annotation reference area; Acquire a calibration object from the image to be processed, the calibration object is located at any direction angle of the object to be labeled, the calibration object is a circle or a pentagon located in the image to be processed, and the direction angle of the object to be labeled is one of the following: upper left, upper right, lower left, and lower right; Determine a correction corner point coordinate according to the correction object, wherein the correction corner point coordinate is the corner point coordinate closest to the correction object among the first corner point coordinates; Determining the rotation angle and rotation direction of the image to be processed according to the coordinates of the correction corner point and the direction angle of the correction object; Rotating the image to be processed according to the rotation angle and the rotation direction so that the direction of the text information in the object to be annotated is consistent with the direction of the text information in the annotation reference area; Determine a transformation matrix according to the first corner point coordinates and the second corner point coordinates; The text position information in the object to be annotated is determined according to the transformation matrix and the text position information in the annotation reference area, and the text is annotated.

2. The method according to claim 1, characterized in that: The step of determining the coordinates of the correction corner points according to the correction object comprises: Obtaining the centroid coordinates of the calibration object; Determine the coordinates of the corner point closest to the centroid coordinates according to the distance between the centroid coordinates of the calibration object and the first corner point coordinates; The coordinates of the corner point with the closest distance are used as the coordinates of the correction corner point.

3. The method according to claim 1, characterized in that The obtaining of the calibration object from the image to be processed comprises: The image to be processed is preprocessed to obtain a correction object.

4. The method according to claim 3, characterized in that The preprocessing of the image to be processed to obtain a correction object comprises: Convert the image to be processed from RGB color space to HSV color space to obtain a processed image; An image having a single color is extracted from the processed image to obtain a correction object.

5. The method according to claim 3, characterized in that: The preprocessing of the image to be processed to obtain a correction object comprises: Converting the image to be processed into a grayscale image and performing denoising and binarization processing to obtain a processed image; The contour is extracted from the processed image to determine the correction object.

6. The method according to claim 5, characterized in that Extracting the contour in the processed image and determining the correction object comprises: extracting contours from the processed image according to a contour function; determining a target contour from the extracted contours according to a first preset condition; A correction object is obtained based on the target contour.

7. The method according to claim 1, characterized in that The step of determining a target area where the object to be labeled is located in the image to be processed includes: Converting the image to be processed into a grayscale image and performing denoising and binarization processing to obtain a processed image; extracting contours from the processed image according to a contour function; A target area where the object to be marked is located is determined from the extracted contour according to the second preset condition.

8. A text position marking device, characterized in that: include: A first determining unit is used to determine a target area where an object to be annotated is located and a first corner point coordinate corresponding to the target area in the image to be processed, wherein the object to be annotated includes text information; An acquiring unit, used for acquiring the coordinates of a second corner point of the marking reference area and text position information in the marking reference area; A calibration object acquisition unit is used to acquire a calibration object from the image to be processed, wherein the calibration object is located at any direction angle of the object to be labeled, the calibration object is a circle or a pentagon located in the image to be processed, and the direction angle of the object to be labeled is one of the following: upper left, upper right, lower left, and lower right; a correction corner point coordinate determining unit, configured to determine the correction corner point coordinates according to the correction object, wherein the correction corner point coordinates are the corner point coordinates that are closest to the correction object among the first corner point coordinates; a correction unit, configured to correct the image to be processed according to the correction corner point coordinates so that the text information in the object to be annotated is in the same direction as the text information in the annotation reference area; A second determining unit, configured to determine a transformation matrix according to the first corner point coordinates and the second corner point coordinates; A labeling unit, used to determine the text position information in the object to be labeled and label it according to the transformation matrix and the text position information in the labeling reference area; The correction unit is configured to: determine the rotation angle and rotation direction of the image to be processed according to the correction corner point coordinates and the direction angle of the correction object; and rotate the image to be processed according to the rotation angle and the rotation direction.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text position marking method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the text position marking method as described in any one of claims 1 to 7.

Images