Image detection methods, devices and storage media

Abstract

This invention provides an image detection method, device, and storage medium. The method includes: acquiring an image to be detected, the image containing multiple objects; identifying multiple marker points corresponding to the multiple objects in the image; determining multiple detection boxes corresponding to the multiple objects in the image based on the multiple marker points; and determining the association relationship of the multiple objects based on the correspondence between the multiple detection boxes and the multiple marker points, and the distance between the marker points corresponding to different detection boxes. In this scheme, the classification of marker points, the learning of distances between marker points of different objects, and the regression of detection boxes based on marker points all demonstrate the extraction of rich semantic information from the image. Based on this rich semantic information, it is possible to accurately parse the structured information of objects in the image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image detection method, device and storage medium. Background Technology

[0002] With the practical application of Optical Character Recognition (OCR) technology, image detection tasks will be required in an increasing number of application scenarios.

[0003] For example, tables can be detected in images containing tables to identify their structure, so that the table in image form can be converted into an editable Excel file, enabling the storage and editing of information contained in the table image.

[0004] To achieve the above objectives, the prerequisite is that the table structure information can be accurately identified in the image. Summary of the Invention

[0005] This invention provides an image detection method, device, and storage medium to accurately parse the structured information of objects in an image.

[0006] In a first aspect, embodiments of the present invention provide an image detection method, the method comprising:

[0007] Acquire an image to be detected, the image containing multiple objects;

[0008] Multiple marker points corresponding to the multiple objects are identified in the image;

[0009] Based on the multiple marker points, multiple detection boxes corresponding to the multiple objects are determined in the image;

[0010] The association relationship of the multiple objects is determined based on the correspondence between the multiple detection boxes and the multiple marker points, as well as the distance between the marker points corresponding to different detection boxes.

[0011] In a second aspect, embodiments of the present invention provide an image detection apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire an image to be detected, wherein the image contains multiple objects;

[0013] The detection module is used to identify multiple marker points corresponding to the multiple objects in the image, determine multiple detection boxes corresponding to the multiple objects in the image based on the multiple marker points, and determine the association relationship of the multiple objects based on the correspondence between the multiple detection boxes and the multiple marker points, and the distance between the marker points corresponding to different detection boxes.

[0014] Thirdly, embodiments of the present invention provide an image detection method, the method comprising:

[0015] Receive a request from a user equipment to call the image detection service interface, the request including an image to be detected, the image containing multiple objects;

[0016] The following steps are performed using the processing resources corresponding to the image detection service interface:

[0017] Multiple marker points corresponding to the multiple objects are identified in the image;

[0018] Based on the multiple marker points, multiple detection boxes corresponding to the multiple objects are determined in the image;

[0019] The association relationship of the multiple objects is determined based on the correspondence between the multiple detection boxes and the multiple marker points, as well as the distance between the marker points corresponding to different detection boxes.

[0020] Fourthly, embodiments of the present invention provide an image detection method, the method comprising:

[0021] Obtain a ticket image containing a table area, wherein multiple cells exist in the table area;

[0022] Multiple marker points corresponding to the multiple cells are identified in the document image;

[0023] Based on the multiple marker points, multiple detection boxes corresponding to the multiple cells are determined in the ticket image;

[0024] Based on the correspondence between the multiple detection boxes and the multiple marker points, and the distance between the marker points corresponding to different detection boxes, the row and column information of the multiple cells in the editable table file is determined;

[0025] An editable table file is generated based on the row and column information.

[0026] Fifthly, embodiments of the present invention provide an electronic device, including: a memory and a processor; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the image detection method as described in the first or fourth aspect.

[0027] In a sixth aspect, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image detection method as described in the first or fourth aspect.

[0028] In the image detection scheme provided in this embodiment of the invention, for an image containing multiple objects, when it is necessary to identify the structured information of multiple objects, this structured information can be reflected as whether there is a certain set relationship between different objects, such as positional adjacency, information matching relationship, etc. First, multiple marker points (such as the center point, boundary points, etc. of the objects) corresponding to these multiple objects are identified in the image, and multiple detection boxes corresponding to these multiple objects are regressed based on these marker points using bounding box regression. These multiple detection boxes are used to roughly represent the corresponding positions of these multiple objects in the image. After obtaining the above multiple discrete detection boxes, the relationship between the multiple detection boxes and the multiple marker points, as well as the distance between the marker points corresponding to different detection boxes, is determined.

[0029] In the above scheme, one or more marker points can be defined according to actual needs. By classifying pixels in the image as marker points, on the one hand, the location regions (detection boxes) corresponding to different objects in the image can be regressed through the marker points; that is, different detection boxes are used to represent the various objects included in the image. On the other hand, based on learning the distance between marker points of different objects, the relationship between multiple objects can be determined by combining the distance between the marker points corresponding to the detection boxes of different objects. In this scheme, the classification of marker points, the learning of the distance between marker points of different objects, and the regression of detection boxes based on marker points all demonstrate the extraction of rich semantic information from the image. Based on this rich semantic information, it is possible to accurately complete the parsing of the structured information of objects in the image. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart of an image detection method provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the composition structure of an image detection model provided in an embodiment of the present invention;

[0033] Figure 3 A flowchart of another image detection method provided in an embodiment of the present invention;

[0034] Figure 4a A schematic diagram illustrating the principle of table image detection provided in an embodiment of the present invention;

[0035] Figure 4b This is a schematic diagram of a table image detection scenario provided in an embodiment of the present invention;

[0036] Figure 5a This is a schematic diagram of the detection box splicing result in a table image provided by an embodiment of the present invention;

[0037] Figure 5b This is a schematic diagram illustrating a display detection frame and the detection frame splicing result provided in an embodiment of the present invention;

[0038] Figure 6 A schematic diagram illustrating the composition structure of another image detection model provided in an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram illustrating a table structure recognition process provided in an embodiment of the present invention;

[0040] Figure 8 A flowchart of another image detection method provided in an embodiment of the present invention;

[0041] Figure 9 A schematic diagram illustrating a text detection process provided in an embodiment of the present invention;

[0042] Figure 10 This is a schematic diagram illustrating the application of an image detection method provided in an embodiment of the present invention;

[0043] Figure 11 A flowchart of another image detection method provided in an embodiment of the present invention;

[0044] Figure 12 A flowchart of another image detection method provided in an embodiment of the present invention;

[0045] Figure 13 A flowchart of another image detection method provided in an embodiment of the present invention;

[0046] Figure 14 A flowchart of another image detection method provided in an embodiment of the present invention;

[0047] Figure 15 This is a schematic diagram of the structure of an image detection device provided in an embodiment of the present invention;

[0048] Figure 16 To and Figure 15 The illustrated embodiment provides a schematic diagram of the electronic device corresponding to the image detection apparatus. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0051] The image detection method provided in this embodiment of the invention can be executed by an electronic device, which can be a terminal device such as a PC, laptop, or smartphone, or a server. The server can be a physical server containing an independent host, a virtual server, a cloud server, or a server cluster.

[0052] The main objective of the image detection method provided in this invention is to perform target detection on the image to be detected, that is, to detect the positions of multiple objects contained in the image and the relationships between the multiple objects. Specifically, the position of a detection box surrounding an object can be used to represent the position of that object in the image to be detected.

[0053] In different application scenarios, the images to be detected will be different, and the multiple objects that need to be detected in the images will also be different.

[0054] For example, in a table detection scenario, the image to be detected is an image containing a table area, and multiple objects are multiple cells contained in the table area. In this case, the purpose of image detection is to detect the position of each of these multiple cells in the image and to determine the positional relationship between these multiple cells.

[0055] For example, in some text detection scenarios, the image to be detected refers to an image containing text content, and multiple objects refer to multiple characters contained in the image. In this case, the purpose of image detection is to detect the corresponding positions of these multiple characters in the image and to determine the adjacency relationship between these multiple characters.

[0056] For example, in some information extraction scenarios, the image to be detected can be an image containing key-value pairs. Multiple objects refer to all the keys and all the values ​​contained in the image. In this case, the purpose of image detection is to detect the positions of these keys and values ​​in the image and to determine the relationship (or correspondence, matching relationship) between these multiple keys and values.

[0057] In practical applications, the images to be detected may be images taken by users themselves, and the image quality is difficult to guarantee. For example, in real-world scenarios, tables in user-captured images may exhibit visual characteristics such as rotation, reflection, overlay, and wrinkles, which poses a greater challenge to image detection tasks. However, the image detection scheme provided in this invention can accurately detect the position and relationships between multiple objects in an image, even if the image to be detected has some visual defects.

[0058] The execution process of the image detection method provided herein is illustrated below with reference to the following embodiments.

[0059] Figure 1 A flowchart of an image detection method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes the following steps:

[0060] 101. Obtain the image to be detected, which contains multiple objects.

[0061] 102. Identify multiple marker points corresponding to multiple objects in an image.

[0062] 103. Based on multiple marker points, determine multiple detection boxes in the image corresponding to multiple objects.

[0063] 104. Determine the association relationship between multiple objects based on the correspondence between multiple detection boxes and multiple marker points, as well as the distance between marker points corresponding to different detection boxes.

[0064] As mentioned above, the image detection tasks differ across application scenarios. In practical applications, the target objects (i.e., the aforementioned objects) to be detected can be defined in advance for each application scenario's corresponding image. For example, in the table detection scenario, for an image containing a table area, the target objects to be detected are the individual cells within the table area. In the text detection scenario, for an image containing text, the target objects to be detected are the individual characters within the image.

[0065] First, for an image containing multiple objects to be detected, it is necessary to detect the positions of these multiple objects in the image.

[0066] Specifically, a pre-trained image detection model can be used to handle the localization of multiple objects. Combined with... Figure 2 The composition structure of this image detection model is illustrated by example.

[0067] like Figure 2 As shown, the image detection model may include a backbone network model and multiple branch network models.

[0068] The backbone network model is used to extract features from the input image to be detected, obtaining a feature map of a certain scale. In practical applications, the backbone network model can optionally be implemented as a convolutional network model consisting of multiple convolutional layers, a residual network model, or other neural network models. Figure 2 As shown, the feature maps output by the backbone network model are input into multiple branch network models.

[0069] From the perspective of their respective functional implementations, the multiple branch network models can be respectively called: the marker classification model, the bounding box regression model, and the concatenation relationship regression model.

[0070] The marker point classification model is used to classify and identify feature points based on the input feature map. The feature map contains several feature points; assuming the spatial resolution of a feature map is h*w, this means that the feature map contains h*w feature points. These feature points have a positional mapping relationship with the pixels in the image to be detected. Therefore, determining the category corresponding to a feature point also determines the category corresponding to the corresponding pixel in the image to be detected.

[0071] In practical applications, one or more category labels for defining marker points are pre-set according to actual needs. For example, the center point and vertices of an object can be defined as marker points, so three category labels can be pre-set: center point, vertex, and others (or background). The feature map received from the backbone network model is used to classify and identify feature points, obtaining the category label corresponding to each feature point. If the category label corresponding to a certain feature point is ultimately determined to be center point or vertex, then the pixel position corresponding to that feature point in the image to be detected is determined. This pixel position is a marker point, specifically corresponding to the center point or vertex of a certain object. The marker point classification model can identify multiple marker points from the image to be detected; these multiple marker points correspond to multiple objects contained in the image.

[0072] In practical applications, assume two categories of markers are predefined: center points and vertices. The marker classification model can include multiple convolutional layers. The input feature map, after passing through these layers, yields a two-channel feature map used to detect these two types of markers: if a feature point is a center point, the corresponding position in the first channel of the feature map is output as 1; if it is a vertex, the corresponding position in the second channel is output as 1 or 0; if it is background, the corresponding positions in all channels are output as 0. Thus, the two channels correspond to the two class labels respectively.

[0073] It should be noted that, firstly, when there is more than one type of marker point defined, the number of marker points identified will be greater than the number of objects; secondly, the classification result only identifies the positions of the numerous marker points included in the image to be detected, and cannot determine the correspondence between the marker points and the objects, that is, it does not know which marker point belongs to which object.

[0074] The detection box regression model is used to regress detection boxes corresponding to multiple objects in the image to be detected based on multiple markers output by the marker classification model. This detection box regression model is trained to learn the distance from the center point of an object to its vertices. Based on this, for a center point output by the marker classification model, the detection box regression model can predict the distance from that center point to the corresponding vertex of the object. Based on this distance, the predicted vertex coordinates of the object can be obtained, and the corresponding detection box can be constructed from these vertex coordinates. The number of vertices of an object is related to its shape; for example, if an object is rectangular, it has four vertex coordinates, and if an object is triangular, it has three vertex coordinates.

[0075] Specifically, the bounding box regression model can include multiple convolutional layers. The feature map input from the backbone network model is processed by multiple convolutional layers to obtain a multi-channel feature map. The number of channels is twice the number of vertices of an object, because a vertex coordinate consists of two coordinates: a horizontal coordinate and a vertical coordinate. The multi-channel feature map output by the bounding box regression model records multiple coordinate values ​​corresponding to each feature point. For a given feature point, one coordinate value corresponding to that channel is recorded in the feature map of one channel. Based on the center point coordinates output by the marker point classification model, the feature point corresponding to the center point coordinates can be determined in this multi-channel feature map. Then, by sequentially querying the multi-channel feature map, multiple coordinate values ​​corresponding to that feature point are obtained. These multiple coordinate values ​​correspond to the distances from the center point to multiple vertices of the corresponding object. Based on these distances, the coordinates of each vertex of the corresponding object can be determined, resulting in the corresponding bounding box.

[0076] The concatenation relationship regression model is used to regress the distance between a marker point of one object and a marker point of another object. This distance is used to identify whether there is a concatenation relationship between bounding boxes corresponding to different objects. The two objects refer to objects with a defined association, such as two adjacent objects or two objects forming a key-value pair. This concatenation relationship regression model is trained to learn the distance between a marker point of an object and the marker points of other objects with a defined relationship. Based on this, for a marker point output by the marker point classification model, the concatenation relationship regression model can predict the distance from that marker point to a target marker point. The object corresponding to the target marker point and the object corresponding to the marker point have a defined association relationship. Therefore, the concatenation relationship can be determined based on this distance. The structure and working principle of this model are similar to those of the bounding box regression model, and will not be elaborated further here. Based on the composition of the image detection model described above, when processing the image to be detected using this model, firstly, a marker classification model identifies multiple markers corresponding to multiple objects in the image. These markers can include various categories of markers, such as object center points and object vertices. Next, based on the identification results of multiple markers, a bounding box regression model determines multiple bounding boxes corresponding to multiple objects in the image. These bounding boxes correspond one-to-one with multiple objects, representing the respective location regions of multiple objects in the image. Then, based on the identification results of multiple markers, a concatenation relationship regression model finally determines the concatenation relationship of the multiple bounding boxes, that is, it determines which bounding boxes are concatenated with which other bounding boxes. If several bounding boxes are determined to be concatenated, it indicates that there is a certain established association between the objects corresponding to these bounding boxes. This achieves the localization of the positions of each object in the image and the identification of the relationships between different objects, which is equivalent to a structured analysis of the information contained in the image, providing the necessary prerequisites for subsequent image processing.

[0077] Optionally, the process of determining the concatenation relationship of multiple detection boxes using a concatenation relationship regression model can be implemented as follows: Based on the correspondence between multiple detection boxes and multiple marker points, and the distances between marker points corresponding to different detection boxes, the concatenation relationship of multiple detection boxes is determined. Specifically, assuming a detection box R is regressed based on a marker point i, after inputting marker point i into the concatenation relationship regression model, based on the model's ability to learn the distances between marker points of different objects, the model can output a set of distance values ​​corresponding to marker point i (the distance of a marker point of another object that has a relationship with the object corresponding to marker point i from marker point i). Based on the position of marker point i, the target position can be calculated using these distance values. Among the multiple marker points obtained from classification, the marker point j that matches the target position is determined; that is, the position of marker point j is closest to the target position. Assuming a detection box P is regressed based on marker point j, then detection boxes R and P can be considered to have a concatenation relationship.

[0078] The above alternatives can be applied to situations where only one type of marker point is defined.

[0079] In the above scheme, the classification of marker points, the learning of the distance between marker points of different objects, and the regression of detection boxes based on marker points all demonstrate the extraction of rich semantic information from the image. Based on this rich semantic information, it is possible to accurately complete the parsing of the structured information of objects in the image and avoid interference such as poor image quality, occlusion, and wrinkles.

[0080] As mentioned above, the image detection scheme provided in this embodiment of the invention can be applied to different application scenarios. The following uses table detection scenario and text detection scenario as examples to illustrate the implementation process of the image detection scheme in these two application scenarios.

[0081] Figure 3 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 3 As shown, the method includes the following steps:

[0082] 301. Obtain the image to be detected. The image is an image containing a table area with multiple cells.

[0083] 302. Identify the center point and vertices of multiple cells in an image.

[0084] 303. For any cell center point, determine the cell vertices that belong to the same cell as the cell center point, wherein the cell vertices that belong to the same cell as the cell center point form a detection box corresponding to the cell center point.

[0085] 304. For any cell vertex, determine at least two cell center points that share the vertex, determine at least two detection boxes corresponding to the at least two cell center points, and determine that the at least two detection boxes have a splicing relationship.

[0086] 305. Update the vertex position corresponding to any cell vertex in the at least two detection boxes to the coordinates of any cell vertex.

[0087] The image detection scheme for images containing table regions (referred to as table images for ease of description) provided in this embodiment of the invention can be achieved by means of... Figure 2 This is implemented using the image detection model provided in [the document / platform / etc.]. The following section combines [the following text / etc.] with... Figure 4a and Figure 4b This example illustrates the table image detection process based on this image detection model. Figure 4a In this context, h and w represent resolution, and h / 4 and w / 4 assume that the table image has been downsampled twice.

[0088] like Figure 4a As shown, after the table image is input into the backbone network model, the feature extraction of the table image can be achieved through the backbone network model to obtain the feature map F1.

[0089] like Figure 4a As shown, when feature map F1 is input into the label classification model, the model outputs feature map F2, as illustrated in the diagram. Feature map F2 is a 2-channel feature map, with each channel corresponding to one of the two types of label points: cell center point and cell vertex. Simply put, feature map F2 describes the category determination result for each feature point: either a cell center point, a cell vertex, or background. In this way, the label classification model can identify multiple cell vertices and multiple cell center points within a table image.

[0090] like Figure 4aAs shown, in a table detection scenario, the detection box regression model regresses the distances from the center point of a cell to the four vertices of its corresponding cell (i.e., cells belonging to the same cell). Since each vertex's coordinates include both x and y coordinates, it outputs eight coordinate values. Therefore, inputting feature map F1 into the detection box regression model will output feature map F3, as shown in the diagram. Feature map F3 is an 8-channel feature map used to describe the eight coordinate values ​​corresponding to each feature point. Based on the coordinates of each cell's center point output by the label point classification model, the feature point corresponding to each cell's center point can be located in feature map F3, thereby obtaining the eight coordinate values ​​corresponding to that feature point, which are the distances from the corresponding cell's center point to the four vertices of its corresponding cell. Based on these distances, the coordinates of these four cell vertices can be determined. For any cell's center point, the cell vertices belonging to the same cell as that cell's center point form the detection box corresponding to that cell's center point. In this way, for each cell's center point, a corresponding detection box is regressed, and the position of the detection box indicates the corresponding cell's position in the table image.

[0091] like Figure 4a As shown, in the table detection scenario, the concatenation relationship regression model is used to regress the distance from a cell vertex to the center point of other cells sharing that vertex. A maximum of four cells can share a single cell vertex, therefore a cell vertex can regress the distance to a maximum of four cell center points. Each cell center point's coordinates include both x and y coordinates, resulting in an output of eight values. Therefore, inputting feature map F1 into the concatenation relationship regression model will output feature map F4, as shown in the diagram. Feature map F4 is an 8-channel feature map used to describe the eight coordinate values ​​corresponding to each feature point. Based on the coordinates of each cell vertex output by the marker point classification model, the feature point corresponding to each cell vertex can be located in feature map F4, thus obtaining the eight coordinate values ​​corresponding to that feature point. For a given cell vertex, these eight coordinate values ​​represent the distance between it and the center points of other cells sharing that vertex. Based on these distances, the coordinates of the four cell center points sharing that vertex can be determined.

[0092] It is understandable that a vertex of a cell can be shared by a maximum of 4 cells. Therefore, in reality, for a vertex of a cell, there may be 2, 4, or 6 coordinate values ​​of 0 out of its corresponding 8 coordinate values. If 4 of the 8 coordinate values ​​of a vertex of a cell are 0, it means that the vertex of that cell is shared by only two cells.

[0093] Suppose that for a certain cell vertex, at least two cell center points that share that cell vertex are finally determined. Then, in the detection boxes obtained based on the detection box regression model corresponding to multiple cell center points, at least two detection boxes corresponding to the at least two cell center points can be determined, thereby determining that the at least two detection boxes have a splicing relationship.

[0094] Then, the detection boxes with splicing relationships can be spliced ​​together, so that multiple detection boxes will eventually be spliced ​​into a complete table, thus obtaining the composition structure of the table in the table image.

[0095] For ease of understanding, combined with Figure 4b The above table image detection process is used as an example to illustrate the various detection results obtained.

[0096] exist Figure 4b In the image, a portion of the table area is indicated by thinner lines, and this area includes four cells. Black dots represent the center points of the cells obtained through the marker point classification model, and black triangles represent the vertices of the cells obtained through the marker point classification model.

[0097] For the four cell center points shown in the figure, the detection box regression model will produce four corresponding detection boxes, denoted as Q1, Q2, Q3, and Q4, which correspond to the four rectangles drawn with bold lines in the figure.

[0098] For a single cell vertex shown in the diagram, an auxiliary box, denoted as Q5, is obtained based on the splicing relationship regression model. The vertices of this auxiliary box Q5 are used to represent the positions of the center points of the cells that share this cell vertex.

[0099] Furthermore, for each vertex of the auxiliary bounding box Q5, among the multiple cell center points obtained through the marker point classification model, the matching cell center point is determined, where "matching" means the closest distance. The determination result is the four cell center points shown in the figure. Finally, the determination result of the four detection boxes corresponding to these four cell center points having a splicing relationship is obtained, and then these four detection boxes are spliced.

[0100] like Figure 4b As shown, during the process of stitching these four detection boxes based on the above cell vertices, the positions of the vertices corresponding to the four detection boxes and the cell vertex are updated to the cell vertex. This is equivalent to dragging the corresponding vertices of these four detection boxes to the cell vertex. The corresponding vertex refers to the vertex on the detection box that is closest to the cell vertex.

[0101] By stitching together the detection boxes based on the determination results of the stitching relationship between the detection boxes, the positional relationship between the objects corresponding to the detection boxes (referring to cells in this embodiment) can be visually and intuitively represented. That is, it can be shown whether there is a certain relationship between different objects contained in the image, because the detection boxes corresponding to the objects with the relationship will be stitched together.

[0102] exist Figure 4b The image only illustrates the stitching effect of detection boxes based on a single cell vertex. In reality, by sequentially traversing each cell vertex output by the marker point classification model, and performing the aforementioned detection box stitching relationship judgment and stitching process for each cell vertex, the multiple discrete (independent) detection boxes output by the detection box regression model will ultimately be stitched into a complete table. It can be understood that the "complete table" here refers to a complete understanding of the overall structure of a table contained within the image; that is, knowing how many cells the table contains and the positional relationships between different cells. Understanding this complete table structure is a prerequisite for converting the image-formatted table into an editable table file, i.e., generating an Excel spreadsheet.

[0103] For ease of understanding, combined with Figure 5a An example illustrating the result of stitching together the detection boxes. Figure 5a In the process, the four originally discrete detection boxes Q1, Q2, Q3, and Q4 will eventually be pieced together to achieve the effect shown in the figure: the detection boxes corresponding to adjacent cells will share a common edge.

[0104] In one optional embodiment, the stitching result of multiple detection boxes can be displayed on a tabular image in a first style according to the stitching relationship of the multiple detection boxes, for user editing. Alternatively, multiple detection boxes obtained based on the detection box regression model can be displayed on the image in a second style, for user editing.

[0105] The system displays multiple bounding boxes obtained based on the bounding box regression model, allowing users to see the position of each bounding box. If a user finds that the position of a bounding box is inaccurate, they can manually adjust it (such as moving or dragging the lines) so that the bounding box regression model can be optimized based on the user's adjustment results.

[0106] Similarly, users can observe the stitching results of multiple detection boxes and identify inaccurate stitching results, then manually adjust them to optimize the stitching relationship regression model based on the user's adjustments.

[0107] The first and second styles can be represented by different colors, different line thicknesses, different line shapes, and so on. For example... Figure 5bAs shown in the figure, the thinner lines represent the initial detection box recognition results, and the thicker lines represent the detection box stitching results. The collected table image is not shown in the figure. In fact, displaying the above detection boxes and detection box stitching results in the table image allows users to intuitively see the accuracy of the detection boxes and detection box stitching results, making it easier to make corresponding adjustments.

[0108] In an optional embodiment, after obtaining multiple detection boxes, the different detection boxes can be displayed differently based on the confidence level of each detection box. The confidence level of each detection box is directly output by the detection box regression model and represents the accuracy of the detection result of a detection box. Differentiating the display of different detection boxes according to their confidence levels can involve displaying detection boxes with confidence levels below a set threshold, allowing users to focus on the boxes with lower confidence levels and make timely corrections; or, alternatively, displaying detection boxes with confidence levels above the set threshold in one style and those with confidence levels below the set threshold in another style, such as lines of different thicknesses or colors.

[0109] The above embodiments take a table detection scenario as an example to illustrate the application of table detection based on... Figure 2 The image detection process performed by the illustrated image detection model is shown in the image below. In fact, this image detection scheme can also be based on... Figure 6 The image detection model shown in the diagram is used to complete this task.

[0110] Figure 6 This is a schematic diagram of the composition structure of another image detection model provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the image detection model provided in this embodiment is similar to... Figure 2 The difference between the image detection model shown in the diagram lies in the addition of a branch network model: the offset regression model. The offset regression model determines the coordinate offsets of each marker point output by the marker point classification model. After performing several convolutional operations on the feature map F1 received from the backbone network model, the offset regression model obtains a 2-channel feature map F5. If a feature point in feature map F5 is a marker point, then the offsets of the horizontal and vertical coordinates of that marker point due to downsampling will be output on this 2-channel feature map F5.

[0111] In practical applications, the backbone network model performs multiple downsampling operations on the input image to be detected during feature extraction. The downsampling operation requires rounding the coordinates of the marker points, which leads to a decrease in the accuracy of the marker point coordinate calculation results. In order to compensate for the accuracy loss caused by downsampling, the error caused by the downsampling operation needs to be compensated back. The aforementioned offset is the error value caused by the downsampling operation.

[0112] The reason for compensating for the offset of the object center point coordinates output by the detection model is that the detection model uses multiple downsampling operations during the layer-by-layer feature extraction process of the image to be detected. The downsampling operation requires rounding the object center point coordinates, which leads to a decrease in the accuracy of the object center point coordinate calculation results. In order to compensate for the accuracy loss caused by downsampling, the error caused by the downsampling operation needs to be compensated back. The aforementioned offset is the error caused by the downsampling operation.

[0113] The training process of the offset regression model can be described as follows: For a training sample image containing a certain object, given the coordinates of the object's markers (such as the center point or vertices), the offset of the object's markers can be calculated based on the downsampling factor of the training sample image by the marker classification model. The calculation formula is as follows:

[0114] x1 = x0 / 2 n -int(x0 / 2 n ), y1=y0 / 2 n -int(y0 / 2 n );

[0115] Where (x0, y0) are the x and y coordinates of the object's marker, respectively, (x1, y1) are the offsets corresponding to the x and y coordinates, int() is the floor function, and n represents the number of times the marker classification model has rounded down the training sample images. n Downsampling processing by a factor of 1.

[0116] With the aforementioned offsets serving as supervisory information, the offset regression model is trained.

[0117] Taking table detection as an example again, based on Figure 6The image detection model shown obtains the coordinates of multiple cell center points and multiple cell vertices through a marker point classification model, and obtains the coordinate offsets of these cell center points and cell vertices through an offset regression model. For a given cell center point, after obtaining the distances between the cell center point and the four vertices of its corresponding cell through a bounding box regression model, the coordinates of the cell center point are updated by adding its coordinate offset. Then, based on the updated coordinates of the cell center point and the aforementioned distances, the coordinates of the four corresponding cell vertices are obtained. Similarly, for a given cell vertex, after obtaining the distances between the vertex and at most four cell center points sharing the same vertex through a concatenation relationship regression model, the coordinates of the cell vertex are updated by adding its coordinate offset. Then, based on the updated coordinates of the cell vertex and the aforementioned distances, the coordinates of the four corresponding cell center points are obtained.

[0118] Taking table detection as an example, after completing the stitching process described above for multiple detection boxes corresponding to multiple cells, we can also perform deep analysis of the table structure based on the stitching results of multiple detection boxes. Deep analysis of the table structure refers to determining the row and column numbers of each cell in the table, so as to convert the image-based table area into an editable table file, such as an Excel spreadsheet.

[0119] In summary, the process involves determining the position information (row and column numbers) of multiple cells within an editable table file based on the concatenation of multiple detection boxes. This position information is then used to generate an editable table file. Specifically, multiple detection boxes are concatenated according to their connection relationships to obtain the vertex positions of each concatenated detection box; based on these vertex positions, the row and column information of the cells within the editable table file is determined; and finally, an editable table file is generated based on this row and column information.

[0120] It is understandable that the stitching result of the above multiple detection boxes is equivalent to marking a complete table area in the image and the position of each cell in the table area. Based on this marking result, the table area is converted into a corresponding editable table file, which makes it convenient for users to store, count, edit and process the data information contained in each cell.

[0121] Combination Figure 7 This example briefly illustrates the process of table structure recognition. For instance... Figure 7As shown in the diagram, assuming the detection box stitching process results in a complete table area, this area is formed by stitching together 6 detection boxes. This yields the vertex positions of each of the stitched detection boxes. Then, based on the vertex positions of the stitched detection boxes, all row lines and column lines can be identified and numbered sequentially: row line 1, row line 2, row line 3, row line 4; and all column lines are numbered sequentially: column line 1, column line 2, column line 3. Based on the row and column line numbering, it can be seen that these row and column lines form a three-row, two-column table. Furthermore, based on the row and column line numbers corresponding to each detection box, the row and column numbers of the detection box can be determined. Figure 7 In the middle, with A ij The row and column numbers are used to determine the results based on the row and column numbers corresponding to each detection box, thus generating the corresponding Excel spreadsheet.

[0122] Figure 8 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 8 As shown, the method includes the following steps:

[0123] 801. Obtain the image to be detected, which contains multiple characters.

[0124] 802. Identify the center points of multiple characters in an image.

[0125] 803. Based on the center points of multiple characters, determine multiple text boxes in the image that correspond to the multiple characters.

[0126] 804. Determine the adjacency relationship of multiple texts based on the correspondence between multiple text boxes and the center points of multiple texts, as well as the distance between the center points of the texts corresponding to different text boxes.

[0127] 805. Extract the target image region from the image, perform text recognition processing on the target image region to obtain the corresponding text content, wherein the target image region is composed of at least two text boxes spliced ​​together.

[0128] This embodiment uses a text detection scenario as an example to illustrate an optional implementation process of the image detection method provided by this invention in a text detection scenario. In this case, the image to be detected is an image containing multiple characters, and the marker points mentioned above can be the center points of the characters.

[0129] After obtaining multiple text center points in an image through a marker point classification model, the distance from each text center point to the vertex of its corresponding text box can be obtained based on a detection box regression model, thus obtaining a corresponding text box for each text center point.

[0130] For ease of understanding, in combination with Figure 9 as an example, in Figure 9 assuming that an image includes the two characters "milk powder", the text center points k and p of the text can be obtained through the identification point classification model, and it is assumed that the text boxes corresponding to these two text center points are respectively represented as W1 and W2.

[0131] In addition, for any text center point k output by the identification point classification model, the distance between the text center point k and the center point of its corresponding adjacent text can also be determined through the splicing relationship regression model. Based on this distance, the center point coordinates of the adjacent text corresponding to the text center point k can be determined. Assuming that the text center point corresponding to this center point coordinates is determined as the text center point p from the multiple text center points output by the identification point classification model, in this way, it can be determined that there is a splicing relationship between the text box W1 corresponding to the text center point k and the text box W2 corresponding to the text center point p. This splicing relationship can reflect that the texts corresponding to these two text boxes are in an adjacent position relationship, and may be two characters in a word or a sentence. Splice these two text boxes with a splicing relationship together to obtain a target image area. Then, the target image area can be intercepted from the image, and the target image area is input into the text recognition model for text recognition processing of the target image area to obtain the corresponding text content: milk powder. At this time, the splicing of the boxes can be realized as: merging the two adjacent boundary lines in the middle of two adjacent text boxes into one, or generating an outer box that includes the adjacent text boxes as the splicing result, as shown in Figure 9 shown in

[0132] As described above, the image detection method provided by the present invention can be executed in the cloud. A number of computing nodes can be deployed in the cloud, and each computing node has processing resources such as computing and storage. In the cloud, a certain service can be organized to be provided by multiple computing nodes. Of course, a computing node can also provide one or more services. The way for the cloud to provide this service can be to provide a service interface externally, and users call this service interface to use the corresponding service. The service interface includes forms such as a Software Development Kit (SDK for short) and an Application Programming Interface (API for short).

[0133] For the solution provided in the embodiment of the present invention, the cloud can provide a service interface for the image detection service. The user calls this image detection service interface through the user device to trigger a request to call this image detection service interface to the cloud. The cloud determines the computing node that responds to this request, and uses the processing resources in this computing node to execute the following steps:

[0134] Receive a request from a user equipment to call the image detection service interface, the request including an image to be detected, the image containing multiple objects;

[0135] The following steps are performed using the processing resources corresponding to the image detection service interface:

[0136] Multiple marker points corresponding to the multiple objects are identified in the image;

[0137] Based on the multiple marker points, multiple detection boxes corresponding to the multiple objects are determined in the image;

[0138] The association relationship of the multiple objects is determined based on the correspondence between the multiple detection boxes and the multiple marker points, as well as the distance between the marker points corresponding to different detection boxes.

[0139] The detailed process of the image detection service interface performing image detection processing using processing resources can be found in the relevant descriptions in the other embodiments mentioned above, and will not be repeated here.

[0140] For ease of understanding, combined with Figure 10 To illustrate with an example. In Figure 10 In this context, when a user wants to perform detection processing on an image, they call the image detection service interface in user device E1 to send a call request to cloud computing node E2. This call request includes the image to be detected and may also include information about the type of the object to be detected. Figure 10 The illustrated method for calling the image detection service interface is as follows: A user uses a specific app with an "Upload" button on one of its interfaces. The user loads the image to be detected on this interface and clicks the upload button, triggering the aforementioned call request. In other words, this app is a client program providing image detection services in the cloud, and the upload button within this program is the application programming interface (API) for calling this service. After loading the original image to be detected, the user can also perform various image editing operations on the image using the various image editing tools provided under the "Image Editing" menu, such as scaling, cropping, and other preprocessing, to enhance image quality.

[0141] In this embodiment, it is assumed that after receiving the call request, the cloud computing node E2 determines, based on the aforementioned information, which type of object needs to be detected in the image to be detected, and then executes the detection process. The detection process is described in the previous embodiment and will not be repeated here. By executing the image detection scheme described above, the cloud computing node E2 can determine the positions of the multiple objects in the image to be detected (i.e., the positions of multiple detection boxes) and the relationships between them (represented by the stitching relationship between detection boxes). Optionally, the cloud computing node E2 can feed back these detection results to the user device E1 so that the user device E1 can perform subsequent processing based on the detection results, such as detection box stitching, table structure recognition, and text recognition as described above. Alternatively, after obtaining the above detection results, the cloud computing node E2 can further perform subsequent processing on the image to be detected based on the detection results, such as detection box stitching, table structure recognition, and text recognition as described above, and feed back the final processing result to the user device E1.

[0142] For ease of understanding, in Figure 10 The following is an example of an application scenario: the image uploaded by the user to be detected is an image taken from a taxi ticket, such as... Figure 10 As shown, the taxi receipt contains multiple key-value pairs in a key:value format. Based on the image detection process described above for text detection scenarios, each key and value can be considered a word (a word is equivalent to a character in the previous text, or it can also be described as a text block). After obtaining all the text boxes corresponding to all keys and values, the concatenation relationship between the text boxes corresponding to the keys and the text boxes corresponding to the values ​​is further determined. This concatenation relationship reflects the relationship between the key and the value, i.e., which key and which value form a key-value pair. Based on the recognition results of the text content contained in each text box and the determination results of the key-value pair relationship, the key-value information included in the taxi receipt image can be recorded in document form, resulting in a structured output of the information. In practical applications, for example, financial personnel can extract relevant information from it based on reimbursement needs to complete the reimbursement process.

[0143] like Figure 10 As shown in the example above, the taxi receipt image and the information extraction results fed back by the computing node E2 can be displayed simultaneously on the user device E1: an information structure consisting of at least one set of key-value pairs. The user can compare and view these results to identify any errors and make corrections.

[0144] In practical applications, image detection is required in many fields, and the technical solutions of the embodiments of the present invention can be used in all of them. The following are examples of several embodiments.

[0145] Figure 11 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 11 As shown, the method may include the following steps:

[0146] 1101. Obtain a ticket image containing a table area, which contains multiple cells.

[0147] 1102. Identify multiple marker points corresponding to multiple cells in a ticket image.

[0148] 1103. Based on multiple marker points, determine multiple detection boxes corresponding to multiple cells in the ticket image.

[0149] 1104. Based on the correspondence between multiple detection boxes and multiple marker points, and the distance between the marker points corresponding to different detection boxes, determine the row and column information of multiple cells in an editable table file.

[0150] 1105. Generate an editable table file based on the row and column information.

[0151] The solution provided in this embodiment can be applied to invoice recognition scenarios. In this application scenario, it is assumed that the invoice is a table-based invoice, such as a general invoice, a value-added tax invoice, various reports, a statement of account, etc.

[0152] In this application scenario, the ultimate goal of image detection on invoice images is to convert the image-formatted table into an editable table file (such as an Excel spreadsheet), and then fill the corresponding data content contained in the table in the image into the editable table file to facilitate the storage, editing, statistical analysis, and other processing of the table data.

[0153] As described above in the embodiments related to table detection scenarios, the multiple marker points to be identified in this embodiment can be either cell vertices or cell center points. The implementation process of the above-described solution can be found in the descriptions of other related embodiments, and will not be repeated here.

[0154] Alternatively, to allow users to check the accuracy of the converted form file, the invoice image and the final form file can be displayed on the same interface. Users can compare and check the accuracy of the conversion results and correct any errors.

[0155] Additionally, if a table contains a large number of cells, displaying both the invoice image and the table file on the same interface may make it difficult for users to spot erroneous cells. To facilitate user inspection, the following solution can be optionally adopted:

[0156] Based on the confidence levels corresponding to multiple detection boxes, determine the target detection boxes whose confidence levels meet the set requirements from the multiple detection boxes;

[0157] The same interface displays the ticket image and the table file, where the text content corresponding to the target detection box is displayed in the table file in a set style.

[0158] The aforementioned multiple detection boxes are detection boxes corresponding to multiple cells in the table area. The detection box regression model described above can be used to detect multiple detection boxes. In practical applications, when this detection box regression model outputs the eight coordinate values ​​corresponding to the center point of a cell (i.e., the distances from the center point to the four vertices of its corresponding cell), it also outputs a confidence score. This confidence score represents the probability that the distances from the center point to the four vertices of its corresponding cell are these eight coordinate values. This confidence score can be used as the confidence score of the detection box corresponding to the center point of that cell. A threshold can be set. If the confidence score of a detection box is lower than this threshold, it indicates that the recognition result of that detection box may be incorrect. This detection box is then highlighted as the target detection box in the generated table file, allowing users to focus on potentially incorrect cells.

[0159] Figure 12 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 12 As shown, the method may include the following steps:

[0160] 1201. Obtain a product image, which includes multiple texts.

[0161] 1202. Identify the center points of multiple texts contained in a product image.

[0162] 1203. Based on the center points of multiple texts, determine multiple text boxes in the product image that correspond to the multiple texts.

[0163] 1204. Determine the adjacency relationship of multiple texts based on the correspondence between multiple text boxes and the center points of multiple texts, as well as the distance between the center points of the texts corresponding to different text boxes.

[0164] 1205. Extract the target image region from the product image, perform text recognition processing on the target image region to obtain the corresponding text content, wherein the target image region is composed of at least two text boxes spliced ​​together.

[0165] 1206. Determine whether the text content contains sensitive words.

[0166] The solution provided in this embodiment is applicable to e-commerce scenarios, where merchants upload product images containing a lot of text information, such as product names and descriptions. Text recognition processing can be performed on the product images to extract the text content. This text recognition is based on determining the location of the text within the product image. The text location is represented by a text box.

[0167] In the solution provided in this embodiment, by detecting the text box corresponding to each character, and then concatenating adjacent text boxes based on the determination of the splicing relationship between the text boxes, the target image area occupied by the spliced ​​text boxes is located in the product image. The target image area is used as the input for a text recognition process to obtain the text content contained therein. If several text boxes are spliced ​​together, it means that the text corresponding to these text boxes is likely to form a word or a sentence. Treating them as a whole for text recognition processing also helps improve the accuracy of the text recognition results.

[0168] The detailed process of text detection described above can be found in the relevant descriptions in the other embodiments mentioned above, and will not be repeated here.

[0169] Afterwards, the extracted text content can be processed according to different application purposes.

[0170] For example, e-commerce platforms need to review whether the text content meets requirements, such as whether it contains sensitive words. In practical applications, a sensitive word database can be pre-built. If words in the sensitive word database are identified in the product image, the product image is considered unsuitable for publication, and the merchant is given a corresponding warning message.

[0171] For example, the category of a product image can be determined based on keywords contained in the text content. The recognized text content may contain product descriptions and other identifying information such as product names. If pre-defined keywords for category classification can be extracted from this information, such as "shoes," "hats," and "skirts," then product category classification can be achieved based on the extracted keywords.

[0172] Figure 13 A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 13 As shown, the method may include the following steps:

[0173] 1301. Obtain the teaching image, which includes multiple texts.

[0174] 1302. Identify multiple center points of text contained in teaching images.

[0175] 1303. Based on the center points of multiple characters, determine multiple text boxes corresponding to multiple characters in the teaching image.

[0176] 1304. Determine the adjacency relationship of multiple texts based on the correspondence between multiple text boxes and the center points of multiple texts, as well as the distance between the center points of the texts corresponding to different text boxes.

[0177] 1305. Extract the target image region from the teaching image, perform text recognition processing on the target image region to obtain the corresponding text content, wherein the target image region is composed of at least two text boxes spliced ​​together.

[0178] 1306. Perform teaching image search processing based on the text content.

[0179] The solution provided in this embodiment can be applied to educational scenarios. In educational scenarios, teachers may use presentation tools such as blackboards and PPTs during the teaching process. Students can take pictures of the teaching images using these presentation tools. When students take a large number of teaching images, they face the need to classify, organize, and retrieve these images as needed.

[0180] When a student needs to search for teaching images related to a specific knowledge point from a large number of collected teaching images, the image detection scheme provided in this invention can be used to perform text detection processing on each collected teaching image. First, multiple text boxes are detected in each teaching image. Then, based on the splicing relationship between the text boxes, the text boxes with splicing relationships are spliced ​​together to form a target image region. Next, text recognition processing is performed on the target image region to obtain the text content it contains. Then, using the desired knowledge point name as the search keyword and the text content recognized in each teaching image as the search library, teaching images containing that knowledge point can be searched.

[0181] In educational settings, besides the aforementioned need for image detection of teaching images, image detection processing can also be applied to teaching materials such as student assignments and test papers. For example, if parents want to collect numerous test questions for summarizing and using as a reference when creating tests for their children, they can photograph their children's assignments and test papers to obtain corresponding images, or they can collect images of assignments and test papers online. Then, the text content, i.e., the test question content, can be identified from the images using the detection scheme described above.

[0182] Figure 14A flowchart of another image detection method provided in an embodiment of the present invention is shown below. Figure 14 As shown, the method may include the following steps:

[0183] 1401. Acquire a medical image containing multiple texts.

[0184] 1402. Identify multiple center points of text contained in medical images.

[0185] 1403. Based on the center points of multiple texts, determine multiple text boxes corresponding to multiple texts in a medical image.

[0186] 1404. Determine the adjacency relationship of multiple texts based on the correspondence between multiple text boxes and the center points of multiple texts, as well as the distance between the center points of texts corresponding to different text boxes.

[0187] 1405. Extract a target image region from a medical image, perform text recognition processing on the target image region to obtain the corresponding text content, wherein the target image region is composed of at least two text boxes spliced ​​together.

[0188] 1406. Perform medical image search processing based on the text content.

[0189] The solution provided in this embodiment can be applied to medical scenarios, where a large number of medical record images and medical images (such as various contrast images) can be generated, and image detection processing can be performed on these images.

[0190] For example, when an institution needs to perform statistical analysis of medical records, it can use the image detection scheme provided in this embodiment of the invention to perform text detection processing on the collected medical images to obtain the text content contained therein. Then, based on the text content corresponding to each medical image, it searches for medical images that match set keywords, such as a certain disease, time period, etc.

[0191] The text detection process for medical images can be implemented by referring to the detection process described in the other embodiments above, and will not be repeated here.

[0192] The above provides illustrative examples of the image detection scheme provided by the embodiments of the present invention applied in some text detection scenarios. In fact, the image detection scheme provided by the embodiments of the present invention can also be used in some table detection scenarios. For example, in e-commerce scenarios, table structure recognition can be performed on product images containing tables; in medical scenarios, table structure recognition can be performed on medical images containing tables; and in educational scenarios, table structure recognition can be performed on exam paper images containing tables. The specific implementation methods for table structure recognition in the above three fields are illustrated below:

[0193] (I) E-commerce scenario:

[0194] A1. Obtain product images containing a table area, where the table area includes multiple cells.

[0195] The product image can be a photograph of the product's outer packaging, or a promotional image designed by the merchant to advertise the product, and so on.

[0196] A2. Identify multiple marker points corresponding to multiple cells in the product image, determine multiple detection boxes corresponding to multiple cells in the product image based on the multiple marker points, and determine the row and column information of multiple cells in an editable table file based on the correspondence between multiple detection boxes and multiple marker points, as well as the distance between marker points corresponding to different detection boxes.

[0197] As mentioned earlier, multiple marker points can be multiple cell vertices and multiple cell center points identified from the table area. The processes of marker point classification, bounding box regression, and distance prediction between marker points in adjacent cells can be implemented using various models provided in the aforementioned embodiments.

[0198] A3. Generate an editable table file based on the row and column information.

[0199] A4. The text content extracted from each detection box will be filled into the corresponding cells in the table file.

[0200] (II) Medical Scenarios:

[0201] B1. Obtain a medical image containing a table area, which includes multiple cells.

[0202] The medical image can be a medical record image, a medical imaging image, etc.

[0203] B2. Identify multiple marker points corresponding to multiple cells in a medical image, determine multiple detection boxes corresponding to multiple cells in the medical image based on the multiple marker points, and determine the row and column information of multiple cells in an editable table file based on the correspondence between multiple detection boxes and multiple marker points, as well as the distance between the marker points corresponding to different detection boxes.

[0204] As mentioned earlier, multiple marker points can be multiple cell vertices and multiple cell center points identified from the table area. The processes of marker point classification, bounding box regression, and distance prediction between marker points in adjacent cells can be implemented using various models provided in the aforementioned embodiments.

[0205] B3. Generate an editable table file based on the row and column information.

[0206] B4. The text extracted from each detection box will be filled into the corresponding cells in the table file.

[0207] (III) Educational Scenarios:

[0208] C1. Obtain a teaching image containing a table area, which includes multiple cells.

[0209] The teaching images can be taken from exam papers, teacher presentation slides or blackboard writing, textbook documents, student assignments, and so on. The images will contain table areas; for example, tables may be present in exam papers, the answer or question areas of assignments, or textbooks.

[0210] C2. Identify multiple marker points corresponding to multiple cells in the teaching image, determine multiple detection boxes corresponding to multiple cells in the teaching image based on the multiple marker points, and determine the row and column information of multiple cells in the editable table file based on the correspondence between multiple detection boxes and multiple marker points, as well as the distance between the marker points corresponding to different detection boxes.

[0211] As mentioned earlier, multiple marker points can be multiple cell vertices and multiple cell center points identified from the table area. The processes of marker point classification, bounding box regression, and distance prediction between marker points in adjacent cells can be implemented using various models provided in the aforementioned embodiments.

[0212] C3. Generate an editable table file based on the row and column information.

[0213] Teachers, parents, and students can modify the content of some cells in the generated table file (which can be an Excel spreadsheet or a table inserted into a document) to achieve purposes such as re-editing the questions.

[0214] The above examples illustrate the application scenarios to which the image detection scheme provided in this embodiment of the invention can be applied, but in reality, it is not limited to these examples.

[0215] The image detection apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these image detection apparatuses can all be configured using commercially available hardware components through the steps taught in this solution.

[0216] Figure 15This is a schematic diagram of the structure of an image detection device provided in an embodiment of the present invention, as shown below. Figure 15 As shown, the device includes: an acquisition module 11 and a detection module 12.

[0217] The acquisition module 11 is used to acquire the image to be detected, which contains multiple objects.

[0218] The detection module 12 is used to identify multiple marker points corresponding to the multiple objects in the image, determine multiple detection boxes corresponding to the multiple objects in the image based on the multiple marker points, and determine the association relationship of the multiple objects based on the correspondence between the multiple detection boxes and the multiple marker points, and the distance between the marker points corresponding to different detection boxes.

[0219] Optionally, the detection module 12 is specifically used to: determine the splicing relationship of the plurality of detection boxes based on the correspondence between the plurality of detection boxes and the plurality of marker points, and the distance between the marker points corresponding to different detection boxes; wherein, if at least two detection boxes have a splicing relationship, it indicates that there is a set association relationship between the objects corresponding to the at least two detection boxes.

[0220] Optionally, the device further includes: a display module, configured to display the stitching result of the multiple detection boxes on the image in a first style according to the stitching relationship of the multiple detection boxes, for user editing; and / or, to display the multiple detection boxes on the image in a second style, for user editing.

[0221] Optionally, the image is an image containing a table area, and the plurality of objects are plurality of cells existing in the table area. In this case, the device further includes: a table generation module, configured to perform splicing processing on the plurality of detection boxes according to the splicing relationship to obtain the vertex positions corresponding to each of the spliced ​​plurality of detection boxes; determine the row and column information corresponding to the plurality of cells in an editable table file based on the vertex positions corresponding to each of the spliced ​​plurality of detection boxes; and generate the editable table file based on the row and column information.

[0222] Optionally, the plurality of objects are multiple texts. In this case, the device further includes: a text recognition module, used to perform splicing processing on the plurality of detection boxes according to the splicing relationship; to extract a target image region from the image, the target image region being composed of at least two spliced ​​detection boxes; and to perform text recognition processing on the target image region to obtain the corresponding text content.

[0223] Optionally, the detection module 12 can be specifically used to: identify multiple cell center points and multiple cell vertices contained in the image; for any cell center point, determine the cell vertices belonging to the same cell as the cell center point, wherein the cell vertices belonging to the same cell as the cell center point constitute a detection box corresponding to the cell center point; for any cell vertex, determine at least two cell center points sharing the cell vertex; determine at least two detection boxes corresponding to the at least two cell center points; determine that the at least two detection boxes have a splicing relationship, and update the vertex position in the at least two detection boxes corresponding to the cell vertex to the coordinates of the cell vertex.

[0224] Figure 15 The device shown can perform the image detection method provided in the foregoing embodiments. For details of the execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0225] In one possible design, the above Figure 15 The structure of the image detection device shown can be implemented as an electronic device, such as... Figure 16 As shown, the electronic device may include a processor 21 and a memory 22. The memory 22 stores executable code, which, when executed by the processor 21, enables the processor 21 to at least implement the image detection method provided in the foregoing embodiments.

[0226] Optionally, the electronic device may also include a communication interface 23 for communicating with other devices.

[0227] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image detection method provided in the foregoing embodiments.

[0228] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0229] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image detection method, characterized in that, include: Acquire an image to be detected, the image containing multiple objects, and the table region in the image has a visual defect; If the plurality of objects are multiple cells contained in a table area in the image, then the center point corresponding to each of the plurality of objects is identified in the image; For any one of the plurality of objects, the plurality of vertices of the object are determined based on the predicted distance from the center point of the object to the vertex of the object. Based on multiple vertices of any object, a detection box corresponding to any object is determined in the image, and the detection box selects a cell in the table. Based on the correspondence between the detection boxes corresponding to the multiple objects and the multiple marker points, and the distance between the marker points corresponding to different detection boxes, the splicing relationship of the multiple detection boxes is determined. The splicing relationship indicates the association relationship of the multiple objects. The multiple marker points include: multiple cell center points and multiple cell vertices.

2. The method according to claim 1, characterized in that, The method further includes: Based on the stitching relationship of multiple detection boxes, the stitching result of the multiple detection boxes is displayed on the image in a first style for user editing; and / or, The plurality of detection boxes are displayed on the image in a second style for user editing.

3. The method according to claim 1, characterized in that, The method further includes: The multiple detection boxes are spliced ​​together according to the splicing relationship to obtain the vertex positions of each of the spliced ​​detection boxes. Based on the vertex positions of the multiple detection boxes after splicing, determine the row and column information of the multiple cells in the editable table file; The editable table file is generated based on the row and column information.

4. The method according to claim 1, characterized in that, The method further includes: If the multiple objects are multiple characters in the image, then the detection boxes corresponding to the multiple characters are determined; The multiple detection boxes are spliced ​​together according to the splicing relationship of the detection boxes corresponding to the multiple characters; A target image region is extracted from the image, and the target image region is composed of at least two detection boxes stitched together; The target image region is subjected to text recognition processing to obtain the corresponding text content.

5. The method according to claim 3, characterized in that, The step of determining a detection box corresponding to any object in the image based on multiple vertices of any object includes: For any cell center point, determine the cell vertex that belongs to the same cell as the cell center point, wherein the cell vertex that belongs to the same cell as the cell center point forms a detection box corresponding to the cell center point. The method further includes: For any given cell vertex, determine at least two cell center points that share that given cell vertex.

6. The method according to claim 5, characterized in that, The step of determining the splicing relationship of the multiple detection boxes based on the correspondence between the detection boxes corresponding to the multiple objects and the multiple marker points, and the distance between the marker points corresponding to different detection boxes, includes: Determine at least two detection boxes corresponding to the center points of the at least two cells; It is determined that at least two detection frames have a splicing relationship; The step of splicing the multiple detection frames according to the splicing relationship includes: Update the vertex position corresponding to any cell vertex in the at least two detection boxes to the coordinates of any cell vertex.

7. An image detection method, characterized in that, include: Receive a request from a user device to call the image detection service interface. The request includes an image to be detected, which contains multiple objects and has visual defects in a table area. The following steps are performed using the processing resources corresponding to the image detection service interface: If the plurality of objects are multiple cells contained in a table area in the image, then the center point corresponding to each of the plurality of objects is identified in the image; For any one of the plurality of objects, the plurality of vertices of the object are determined based on the predicted distance from the center point of the object to the vertex of the object. Based on multiple vertices of any object, a detection box corresponding to any object is determined in the image; Based on the correspondence between the detection boxes corresponding to the multiple objects and the multiple marker points, and the distance between the marker points corresponding to different objects, the splicing relationship of the multiple detection boxes is determined. The splicing relationship indicates the association relationship of the multiple objects. The multiple marker points include: multiple cell center points and multiple cell vertices.

8. An image detection method, characterized in that, include: Obtain an image of a bill containing a table area, wherein the table area contains multiple cells and the table area has visual defects; Identify the center point corresponding to each of the plurality of cells in the document image; For any cell among the plurality of cells, the plurality of vertices of any cell are determined based on the predicted distance from the center point of any cell to the vertex of any cell; Based on multiple vertices of any given cell, a detection box corresponding to that cell is determined in the ticket image; Based on the correspondence between the detection boxes corresponding to the multiple cells and the multiple marker points, and the distance between the marker points corresponding to different detection boxes, the row and column information of the multiple cells in the editable table file is determined. The multiple marker points include: multiple cell center points and multiple cell vertices. An editable table file is generated based on the row and column information.

9. The method according to claim 8, characterized in that, The step of determining the row and column information of the multiple cells in the editable table file based on the correspondence between the detection boxes corresponding to the multiple cells and the multiple marker points, and the distance between the marker points corresponding to different detection boxes, includes: Based on the correspondence between multiple detection boxes and multiple marker points, and the distance between marker points corresponding to different detection boxes, the splicing relationship of the multiple detection boxes is determined; wherein, if at least two detection boxes have a splicing relationship, it indicates that the cells corresponding to the at least two detection boxes have a positional adjacency relationship. The multiple detection boxes are spliced ​​together according to the splicing relationship to obtain the vertex positions of each of the spliced ​​detection boxes. Based on the vertex positions of the multiple detection boxes after splicing, the row and column information of the multiple cells in the editable table file is determined.

10. An electronic device, characterized in that, include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the image detection method as described in any one of claims 1 to 6.

11. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the image detection method as described in any one of claims 1 to 6.

Images