Image-text matching method, device and equipment and storage medium

Abstract

The image-text matching method comprises the following steps: obtaining an initial image; identifying the initial image, generating a picture selection box for each candidate picture in the initial image, and generating a text selection box for each candidate text in the initial image; selecting a first picture edge point for each picture selection box according to a preset first selection rule, and selecting a first text edge point for each text selection box according to a preset second selection rule; combining the first picture edge point with each first text edge point to obtain a plurality of image-text edge groups; filtering the image-text edge groups according to the coordinate values of the first text edge points; calculating the image-text edge area corresponding to the filtered image-text edge groups, and selecting the candidate text in the image-text edge with the smallest area as the candidate text matched with the picture selection box corresponding to the first picture edge point, thereby improving the image-text matching efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, for example to a picture-text matching method, device, equipment and storage medium. BACKGROUND

[0002] For the type of "write words according to pictures", in the automatic correction process, the student's handwriting in the answer area and the picture given by the question need to be matched based on the detection and recognition of the picture and the text.

[0003] The matching method in the prior art is to use a traversal method to one-by-one match the position of the picture and the position of the handwriting answer area to obtain the final matching result. However, this method has high complexity and low matching efficiency. SUMMARY

[0004] The present application aims to provide a picture-text matching method, device, equipment and storage medium which can improve the efficiency of picture-text matching.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A picture-text matching method is provided, which comprises:

[0007] An initial image containing a plurality of candidate pictures and a plurality of candidate texts is obtained;

[0008] The initial image is recognized, and a picture selection frame is generated for each candidate picture in the initial image, and a text selection frame is generated for each candidate text in the initial image;

[0009] According to a preset first selection rule, a first picture frame point is selected for each picture selection frame, and according to a preset second selection rule, a first text frame point is selected for each text selection frame;

[0010] The first picture frame point and the first text frame point are combined respectively to obtain a plurality of picture-text frame groups;

[0011] The picture-text frame groups whose coordinate values of the first text frame points do not satisfy the preset matching condition are filtered;

[0012] The picture-text frame area corresponding to the filtered picture-text frame groups is calculated, and the candidate text in the picture-text frame with the smallest picture-text frame area is selected as the candidate text matched with the picture selection frame corresponding to the first picture frame point.

[0013] The present application also provides a picture-text matching device, which comprises:

[0014] An image acquisition module is configured to acquire an initial image containing a plurality of candidate pictures and a plurality of candidate texts.

[0015] A box generation module is configured to identify the initial image, generate a picture box for each of the candidate pictures in the initial image, and generate a text box for each of the candidate texts in the initial image.

[0016] A border point selection module is configured to perform first picture border point selection on each of the picture boxes according to a preset first selection rule, and perform first text border point selection on each of the text boxes according to a preset second selection rule.

[0017] A border combination module is configured to combine each of the first picture border points with each of the first text border points to obtain a plurality of picture-text border groups.

[0018] A border filtering module is configured to filter the picture-text border groups whose coordinate values of the first text border points do not satisfy a preset matching condition.

[0019] A text matching module is configured to calculate picture-text border areas of the filtered picture-text border groups, and select a candidate text in a picture-text border with a smallest picture-text border area as a candidate text matched with the picture box corresponding to the first picture border point.

[0020] The application further provides a computer device including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the picture-text matching method when executing the computer program.

[0021] The application further provides a computer readable storage medium storing a computer program, and the computer program implements the steps of the picture-text matching method when executed by a processor.

[0022] The picture-text matching method of the present application calibrates the edge frames of the candidate picture and the candidate text, performs edge frame expansion based on the first picture edge frame point of the candidate picture and the first text edge frame point of the candidate text, thereby obtaining a picture-text edge frame group containing each candidate text, improving the completeness of picture-text matching and avoiding omission of candidate text; the picture-text edge frame group whose coordinate values of the first text edge frame point do not satisfy the preset matching condition is filtered, thereby obtaining a picture-text edge frame group meeting the requirements, and the subsequent area calculation amount can be reduced through filtering; since the farther the distance between the candidate picture and the candidate text, the farther the distance between the corresponding first picture edge frame point and the first text edge frame point, the candidate text in the picture-text edge frame with the smallest area is selected as the candidate text matched with the picture frame corresponding to the first picture edge frame point through area sorting, thereby avoiding the problem of reduced matching efficiency caused by the traversal comparison method, improving the matching efficiency, and ensuring that the finally matched candidate text is the text closest to the candidate picture on paper, thereby improving the matching accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 Flowchart of picture-text matching method of an embodiment

[0024] Figure 2 Specific initial image diagram of an embodiment

[0025] Figure 3 Picture frame and text frame diagram of an embodiment

[0026] Figure 4 Picture-text edge frame group diagram of an embodiment

[0027] Figure 5 Picture-text edge frame group diagram of an embodiment

[0028] Figure 6 Structure diagram of picture-text matching device of an embodiment

[0029] Figure 7 Structure diagram of computer device

[0030] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0032] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" and "comprising" and the like, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, modules, modules, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, modules, components, and / or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can also be present. In addition, the use of "connection" or "coupling" herein also includes wireless connection or wireless coupling. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] It is to be understood that the terms used herein are not intended to limit the present application to a given aspect, but rather the terms are intended to include, where appropriate, equivalents and / or logically equivalent variations thereof. Furthermore, it is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" and "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, modules, modules, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, modules, components, and / or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can also be present. In addition, the use of "connection" or "coupling" herein also includes wireless connection or wireless coupling. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] Reference Figure 1 is a flowchart of a picture-text matching method disclosed by the present application, comprising:

[0035] S1: obtaining an initial image containing a plurality of candidate pictures and a plurality of candidate texts;

[0036] S2: identifying the initial image, generating a picture selection frame for each of the candidate pictures in the initial image, and generating a text selection frame for each of the candidate texts in the initial image;

[0037] S3: performing first picture border point selection on each of the picture selection frames according to a preset first selection rule, and performing first text border point selection on each of the text selection frames according to a preset second selection rule;

[0038] S4: combining the first picture border points and the first text border points respectively to obtain a plurality of picture-text border groups;

[0039] S5: filtering the picture-text border groups whose coordinate values of the first text border points do not satisfy a preset matching condition;

[0040] S6: Calculate the area of ​​the image and text borders corresponding to the filtered image and text border group, and select the candidate text in the image and text border with the smallest area as the candidate text to match the image selection box corresponding to the first image border point.

[0041] As described in step S1 above, refer to Figure 2 This is a schematic diagram of a specific initial image provided in this embodiment. It can be understood that the initial image can be an image captured by a camera and transmitted to a computer, or it can be a screenshot of an electronic document on a computer. The initial image provided in this embodiment... Figure 2 The example uses images captured by a camera, including several alternative images such as pictures of cars, teacups, and animals, as well as alternative texts that match different images, such as "horse," "cat," and "car."

[0042] As described in step S2 above, candidate images can be identified using methods such as border recognition, and candidate text can be identified using OCR (Optical Character Recognition). Once a candidate image is identified, an image selection box is generated for it. In a specific implementation, the image selection box is a rectangular selection box. Since candidate images may be non-rectangular shapes such as polygons or circles, the image selection box should be large enough to encompass most of the candidate image area. Similarly, once candidate text is identified, an image selection box is generated for it. In a specific implementation, the text selection box is also a rectangular selection box. Since some strokes may be prominent during writing, the text selection box should be large enough to encompass most of the candidate text area. (Refer to...) Figure 3 This is a schematic diagram of an image selection box and a text selection box provided in this embodiment.

[0043] As described in step S3 above, after obtaining each selection box, the relative coordinates between each selection box can be obtained based on the coordinates of each point that makes up the selection box. It should be noted that, to avoid discrepancies in subsequent area calculations caused by different selection rules, the same first selection rule is followed when selecting the first image border points, while the same second selection rule is followed when selecting the first text border points; see reference. Figure 3 You can select the top left corner of the image selection box as the first image border point, and its coordinates are ( x 1, y 1) Select the top left corner of the text selection box as the reference text border point, and its coordinates are ( x 1', y 1'); while in the actual selection process ( x 1, y 1) and ( x 1',y 1' ) the selected frame cannot contain most of the alternative text content, in order to make the generated picture-text frame group contain more complete alternative text, the embodiment obtains the first text frame point by referring to the text frame point, that is, the point farthest from the reference text frame point in the text selected frame is selected as the first text frame point, and the coordinates thereof are (x1, y1). x 2', y 2' ) at this time, the first text frame point is also the text frame point farthest from the first picture frame point in the text selected frame.

[0044] As described in step S4 above, the first picture frame point and the first text frame point in each text selected frame are combined to obtain a plurality of picture-text frame groups; each picture-text frame group corresponding to a first picture frame point contains an alternative picture corresponding to the first picture frame point and at least one different alternative text; referring to Figure 4 is a picture-text frame group diagram provided by the embodiment, and the embodiment takes the first alternative picture as an example to obtain 6 picture-text frame groups, and the corresponding coordinate groups are respectively: (x1, y1), x 1, y 1, x 2', y 2' ), x 1, y 1, x 3', y 3' ), x 1, y 1, x 4', y 4' ), x 1, y 1, x 5', y 5' ), x 1, y 1, x 6', y 6' ), x 1, y 1, x 7', y 7' ), and the remaining picture selected frames are processed in the same way as the picture selected frame corresponding to (x1, y1) 1), so the details are not repeated. x 1, y 1).

[0045] In a specific embodiment, in addition to a rectangle, the picture-text frame group can also be set to a circular shape according to actual needs, and when it is a circular shape, the line between the first picture frame point and the first text frame point can be taken as a diameter to obtain a circular picture-text frame group.

[0046] As described in step S5 above, since the distance between the alternative picture and the alternative text will not be too large in actual application, the matching condition can be set as that the horizontal coordinate value of the first text border point and the horizontal coordinate value of the first picture border point are not greater than a preset first distance threshold, and the vertical coordinate value of the first text border point and the vertical coordinate value of the first picture border point are not greater than a preset second distance threshold, so as to preliminarily screen a part of the picture-text border groups that obviously do not meet the matching condition from the distance, and reduce the subsequent calculation amount.

[0047] As described in step S6 above, referring to Figure 4 It can be known that, although x 5' and x 1 in the x axis direction are less than x 2' and x 1 in the x axis direction, y 4' and x 1 in the y axis direction are less than y 3' and y 1 in the y axis direction, but x 5', y 5') is obviously farther than x 2', y 2') from x 1, y 1), and x 4', y 4') is obviously farther than x 3', y 3') from x 1, y 1), so it can be seen that the selection of the alternative text that best matches the alternative picture cannot be achieved simply by the distance; therefore, the embodiment adopts the way of calculating the picture-text border area of each picture-text border group, takes the alternative text in the picture-text border group with the smallest area as the alternative text corresponding to the corresponding alternative picture, and thus completes the picture-text matching.

[0048] In summary, by calibrating the bounding boxes of the alternative pictures and the alternative texts, the bounding box expansion is performed based on the first picture bounding box point of the alternative picture and the first text bounding box point of the alternative text, so as to obtain a group of picture-text bounding boxes containing each alternative text, improve the integrity of picture-text matching, and avoid missing of the alternative text; the group of picture-text bounding boxes whose coordinate values of the first text bounding box point do not satisfy the preset matching condition is filtered, so as to obtain a group of picture-text bounding boxes meeting the requirements, and the subsequent area calculation amount can be reduced through the filtering; since the distance between the alternative picture and the alternative text is farther, the distance between the corresponding first picture bounding box point and the first text bounding box point is also farther, so the alternative text in the picture-text bounding box with the smallest area is selected as the alternative text matched with the picture selected box corresponding to the first picture bounding box point through the area sorting, the problem of low matching efficiency caused by the traversal comparison method is avoided, the matching efficiency is improved, and it is ensured that the final matched alternative text is the text closest to the alternative picture on the paper, so the matching accuracy is improved.

[0049] In one embodiment, the first picture bounding box point of each picture selected box is selected according to a preset first selection rule, and the first text bounding box point of each text selected box is selected according to a preset second selection rule, including:

[0050] The coordinate origin is selected in the initial image, and the coordinate system is established;

[0051] According to the coordinate system, the point in the picture selected box closest to the coordinate origin is selected as the first picture bounding box point;

[0052] The point in the text selected box farthest from the first picture bounding box point is selected as the first text bounding box point.

[0053] As described above, when the picture selected box or the text selected box is a parallelogram, a circle or other polygon in a specific embodiment, it is difficult to select the first picture bounding box point according to the vertex of the selected box, therefore, the point in the picture selected box closest to the coordinate origin is selected as the first picture bounding box point, and the point in the text selected box farthest from the first picture bounding box point is selected as the first text bounding box point, so that the first picture bounding box point and the first text bounding box point can contain as much area of the alternative picture and the alternative text as possible, thereby improving the accuracy of subsequent alternative text matching according to the area.

[0054] In one embodiment, when the text selected box is a rectangular selected box, when the picture selected box and the text selected box are both rectangular selected boxes, the first picture bounding box point of each picture selected box is selected according to a preset first selection rule, and the first text bounding box point of each text selected box is selected according to a preset second selection rule, including:

[0055] selecting a top vertex on one side of the upper part of the picture selection frame as the first picture frame point;

[0056] selecting a top vertex on the other side of the first picture frame point and on the lower part of the text selection frame as the first text frame point.

[0057] As described above, when the picture selection frame and the text selection frame are both rectangular frames, the first text frame point can be determined according to the position of the first picture frame point. Specifically, in order to conform to the calculation habits of computers, the left upper corner of the initial image is selected as the coordinate origin, and the point in the picture selection frame closest to the coordinate origin is selected as the first picture frame point, that is, the top vertex on the upper part and on the left side of the picture selection frame. At this time, the top vertex on the lower part and on the right side of the text selection frame is selected as the first text frame point, that is, the point in the text selection frame farthest from the first picture frame point.

[0058] In one embodiment, when the preset matching direction between the candidate picture and the candidate text is the same as the positive direction of the coordinate axis of the coordinate system, the matching condition comprises:

[0059] The absolute value of the horizontal coordinate value of the first text frame point is not less than the absolute value of the horizontal coordinate value of the first picture frame point, and the absolute value of the vertical coordinate value of the first text frame point is not less than the absolute value of the vertical coordinate value of the first picture frame point.

[0060] As described above, the preset matching direction between the candidate picture and the candidate text refers to the component direction of the direction of each candidate picture towards the corresponding candidate text on the x-axis or y-axis. For example, if the candidate text is located on the right lower side of the candidate picture, the direction of the candidate picture towards the corresponding candidate text is the right lower side, and the components of the direction on the x-axis or y-axis are right and down, respectively. As long as one component on the coordinate axis is the same as the positive direction of the coordinate axis, it is considered that the matching direction between the candidate picture and the candidate text is the same as the positive direction of the coordinate axis. For reference Figure 5 At this time, the coordinate origin is in the left upper corner, and the positive direction of the x-axis is towards the right side and the positive direction of the y-axis is towards the lower side. The matching direction is from top to bottom. If the position corresponding to the current picture selection frame is not the left upper corner in the entire initial image, it is possible that the picture-text frame group shown in Figure 5 x 3, y 3, x 2', y 2', x 7, y 7, x 3', y 3', x 7, y 7,​x 4', y In case 4'), the selection box obviously does not contain candidate text. Therefore, this type of image and text selection box group also needs to be filtered out. Therefore, in this embodiment, the first text border point that satisfies the absolute value of the horizontal coordinate value not less than the absolute value of the horizontal coordinate value of the first image border point and the absolute value of the vertical coordinate value not less than the absolute value of the vertical coordinate value of the first image border point is set as the point that meets the matching condition. This filters out the first text border point whose absolute value of the horizontal coordinate value is less than the absolute value of the horizontal coordinate value of the first image border point, or whose absolute value of the vertical coordinate value of the first text border point is less than the absolute value of the vertical coordinate value of the first image border point. This makes each first text border point located at the lower right corner of the first image border point. That is, each filtered image and text border group can contain at least one complete image selection box and at least one complete text selection box.

[0061] In one embodiment, when the origin of the coordinate system is the top-left corner of the initial image and the image selection box is a rectangular selection box, selecting the point in the image selection box closest to the origin of the coordinate system as the first image border point includes:

[0062] Select the vertex located at the top left corner of the image selection box as the first image border point;

[0063] Selecting the point furthest from the first image border point within the text selection box as the first text border point includes:

[0064] Select the lower right corner vertex of the text selection box as the first text border point.

[0065] As described above, to align with computer calculation habits, this embodiment selects the top-left corner of the initial image as the origin of the coordinate system, and each coordinate value is obtained with reference to the origin. The point closest to the origin in the image selection box is selected as the first image border point, i.e., the top-left vertex of the image selection box. Similarly, the point closest to the origin in the text selection box is selected as the first text border point, i.e., the top-left vertex of the text selection box. Then, the bottom-right vertex of the text selection box, i.e., the point diagonally opposite to the first text border point, is selected as the first text border point. This first text border point is also the point in the text selection box that is furthest from the first image border point.

[0066] In one embodiment, calculating the area of ​​the image and text border corresponding to the filtered image and text border group includes:

[0067] Calculate the area of ​​the border of the image / text using a NumPy matrix.

[0068] As mentioned above, Python's NumPy library provides matrix operation functionality, which includes a matrix library called numpy.matlib. NumPy functions not only use the C language but also BLAS (Basic Linear Algebra Subprograms). The BLAS implementation is highly optimized for each operation, such as using the AVX vector instruction set. Therefore, using NumPy matrices to calculate the area of ​​the border of an image or text can effectively improve the efficiency of area calculation.

[0069] In one embodiment, calculating the area of ​​the image / text border using a NumPy matrix includes:

[0070] Using the NumPy matrix, the area of ​​the image and text border corresponding to the filtered image and text border group is calculated using the following formula:

[0071] S=( y n '- y m )*( x n '- x m )

[0072] In the formula, S is the area of ​​the image / text border. y m The ordinate value of the border point of the first image. y n ' represents the ordinate value of the first text border point. x m The x-coordinate value of the border point of the first image. x n ' is the x-coordinate value of the first text border point.

[0073] As described above, this embodiment uses coordinate values ​​to calculate the area of ​​each graphic border group. When the graphic border group to be calculated is ( x 1, y 1, x 2', y When 2'), S=( y 2'- y 1)*( x 2'- x 1) This allows for the quick and accurate determination of the area of ​​each graphic border group, improving computational efficiency.

[0074] In summary, the picture-text matching method provided in the embodiments of the present application, by calibrating the bounding boxes of the candidate pictures and the candidate texts, performing bounding box expansion based on the first picture bounding box point of the candidate picture and the first text bounding box point of the candidate text, thereby obtaining a picture-text bounding box group containing each candidate text, improving the completeness of picture-text matching and avoiding missing of candidate texts; by filtering the picture-text bounding box group whose coordinate value of the first text bounding box point does not satisfy the preset matching condition, obtaining a picture-text bounding box group meeting the requirements, and through filtering, the subsequent area calculation amount can be reduced; since the farther the distance between the candidate picture and the candidate text, the farther the distance between the corresponding first picture bounding box point and the first text bounding box point, therefore by area sorting and selecting the candidate text in the picture-text bounding box with the smallest area as the candidate text matched with the picture selection frame corresponding to the first picture bounding box point, the problem of reduced matching efficiency caused by the traversal comparison method is avoided, the matching efficiency is improved, and it is ensured that the finally matched candidate text is the text closest to the candidate picture on paper, thereby improving the matching accuracy.

[0075] Reference Figure 6 The system comprises:

[0076] The image acquisition module 100 is configured to acquire an initial image containing a plurality of candidate pictures and a plurality of candidate texts.

[0077] The selection frame generation module 200 is configured to identify the candidate pictures, generate a picture selection frame for each of the candidate pictures, and generate a text selection frame for each of the candidate texts.

[0078] The bounding box point selection module 300 is configured to select a first picture bounding box point for each of the picture selection frames, select a first text bounding box point for each of the text selection frames, and obtain a first text bounding box point according to the first text bounding box point.

[0079] The bounding box combination module 400 is configured to combine the first picture bounding box point with each of the first text bounding box points to obtain a plurality of picture-text bounding box groups.

[0080] The bounding box filtering module 500 is configured to filter the picture-text bounding box group whose coordinate value of the first text bounding box point does not satisfy the preset matching condition.

[0081] The text matching module 600 is configured to calculate the picture-text bounding box area corresponding to the filtered picture-text bounding box group, and select the candidate text in the picture-text bounding box with the smallest picture-text bounding box area as the candidate text matched with the picture selection frame corresponding to the first picture bounding box point.

[0082] In one embodiment, the bounding box point selection module 300 is further configured to:

[0083] select a coordinate origin and establish a coordinate system in the initial image;

[0084] According to the coordinate system, select a point in the picture frame closest to the coordinate origin as the first picture frame point;

[0085] select the point in the text frame farthest from the first picture frame point as the first text frame point.

[0086] In one embodiment, when the picture frame and the text frame are both rectangular frames, the frame point selection module 300 is also used to:

[0087] select a top point on one side of the upper part of the picture frame as the first picture frame point;

[0088] select a top point on the other side of the first picture frame point located in the lower part of the text frame as the first text frame point.

[0089] In one embodiment, when the preset matching direction between the alternative picture and the alternative text is the same as the positive direction of the coordinate axis of the coordinate system, the matching condition includes:

[0090] The absolute value of the horizontal coordinate value of the first text frame point is not less than the absolute value of the horizontal coordinate value of the first picture frame point, and the absolute value of the vertical coordinate value of the first text frame point is not less than the absolute value of the vertical coordinate value of the first picture frame point.

[0091] In one embodiment, the frame point selection module 300 is also used to:

[0092] select a top point located in the upper left corner of the picture frame as the first picture frame point;

[0093] The selection of the point in the text frame farthest from the first picture frame point as the first text frame point includes:

[0094] select a top point in the lower right corner of the text frame as the first text frame point.

[0095] In one embodiment, the text matching module 600 is also used to:

[0096] Calculate the picture-text frame area using a numpy matrix.

[0097] In one embodiment, the text matching module 600 is also used to:

[0098] Calculate the picture-text frame area corresponding to the filtered picture-text frame group by using the numpy matrix according to the following formula:

[0099] S=( y n ’- y m )*( x n ’- x m )

[0100] In the formula, S is the picture frame area of the text, y m is a vertical coordinate value of a first picture frame point, y n is a vertical coordinate value of the first text frame point, x m is a horizontal coordinate value of the first picture frame point, x n is a horizontal coordinate value of the first text frame point.

[0101] With reference to Figure 7 , the embodiments of the present application also provide a computer device, which can be a server, and the internal structure of the computer device can be as shown in Figure 7 . The computer device comprises a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store text-image matching data and the like. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a text-image matching method. The text-image matching method comprises: identifying an initial image, generating a picture selection frame for each of the candidate pictures in the initial image, and generating a text selection frame for each of the candidate texts in the initial image; selecting a first picture frame point for each of the picture selection frames according to a preset first selection rule, and selecting a first text frame point for each of the text selection frames according to a preset second selection rule

[0102] Those skilled in the art can understand Figure 7 that the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied.

[0103] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement a picture-text matching method, comprising the steps of: obtaining an initial image containing a plurality of candidate pictures and a plurality of candidate texts; identifying the initial image, generating a picture selection box for each of the candidate pictures in the initial image, and generating a text selection box for each of the candidate texts in the initial image; performing first picture border point selection on each of the picture selection boxes according to a preset first selection rule, and performing first text border point selection on each of the text selection boxes according to a preset second selection rule; combining the first picture border points and the first text border points respectively to obtain a plurality of picture-text border groups; filtering the picture-text border groups whose coordinate values of the first text border points do not satisfy a preset matching condition; calculating picture-text border areas corresponding to the filtered picture-text border groups, and selecting a candidate text in a picture-text border with the smallest area as a candidate text matched with the picture selection box corresponding to the first picture border point. It can be understood that the computer readable storage medium in the embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0104] In summary, for the picture-text matching method, system, device and storage medium provided in the embodiments of the present application, the border of the candidate picture and the candidate text is calibrated, the border is expanded based on the first picture border point of the candidate picture and the first text border point of the candidate text, thereby obtaining a picture-text border group containing each candidate text, improving the completeness of picture-text matching and avoiding omission of candidate texts; the picture-text border groups whose coordinate values of the first text border points do not satisfy the preset matching condition are filtered, thereby obtaining a picture-text border group meeting the requirements, and the subsequent area calculation amount can be reduced through filtering; since the distance between the candidate picture and the candidate text is farther, the distance between the corresponding first picture border point and the first text border point is also farther, therefore, the candidate text in the picture-text border with the smallest area is selected as the candidate text matched with the picture selection box corresponding to the first picture border point through area sorting, thereby avoiding the problem of reduced matching efficiency caused by the traversal comparison method, improving the matching efficiency, ensuring that the final matched candidate text is the text closest to the candidate picture on paper, and improving the matching accuracy.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, databases, or other media in this application and in examples used herein, unless specifically stated otherwise, can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0106] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, device, article, or method that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, device, article, or method. Without more limitations, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, device, article, or method that includes the element.

[0107] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method of image-text matching, wherein, The method comprises: acquiring an initial image containing a plurality of alternative pictures and a plurality of alternative texts; identifying the initial image, generating a picture selection frame for each of the alternative pictures in the initial image, and generating a text selection frame for each of the alternative texts in the initial image; selecting a first picture edge point for each of the picture selection frames according to a preset first selection rule, and selecting a first text edge point for each of the text selection frames according to a preset second selection rule; combining the first picture edge point and the first text edge point to obtain a plurality of picture-text edge frame groups; filtering the picture-text edge frame groups whose coordinate values of the first text edge points do not satisfy a preset matching condition; calculating the picture-text edge frame area corresponding to the filtered picture-text edge frame groups, selecting a picture-text edge frame group with the smallest picture-text edge frame area as a minimum edge frame group, and matching the alternative picture and the alternative text corresponding to the minimum edge frame group; the method of selecting a first picture edge point for each of the picture selection frames according to a preset first selection rule, and selecting a first text edge point for each of the text selection frames according to a preset second selection rule comprises: selecting a coordinate origin in the initial image and establishing a coordinate system; selecting a point closest to the coordinate origin in the picture selection frame as the first picture edge point according to the coordinate system; selecting a point farthest from the first picture edge point in the text selection frame as the first text edge point; when a preset matching direction between the alternative picture and the alternative text is the same as the positive direction of the coordinate axis of the coordinate system, the matching condition comprises: the absolute value of the horizontal coordinate value of the first text edge point is not less than the absolute value of the horizontal coordinate value of the first picture edge point, and the absolute value of the vertical coordinate value of the first text edge point is not less than the absolute value of the vertical coordinate value of the first picture edge point.

2. The graph matching method of claim 1, wherein, when the picture selection frame and the text selection frame are both rectangular frames, the method of selecting a first picture edge point for each of the picture selection frames according to a preset first selection rule, and selecting a first text edge point for each of the text selection frames according to a preset second selection rule comprises: selecting a top point on one side of the upper part of the picture selection frame as the first picture edge point; selecting a top point on the other side of the first picture edge point in the lower part of the text selection frame as the first text edge point.

3. The graph matching method of claim 1, wherein, when the coordinate origin is the top-left corner point of the initial image and the picture selection frame is a rectangular frame, the method of selecting a point closest to the coordinate origin in the picture selection frame as the first picture edge point according to the coordinate system comprises: selecting a top point in the top-left corner of the picture selection frame as the first picture edge point; the method of selecting a point farthest from the first picture edge point in the text selection frame as the first text edge point comprises: selecting a top point in the lower-right corner of the text selection frame as the first text edge point.

4. The graph matching method of claim 1, wherein, the method of calculating the picture-text edge frame area corresponding to the filtered picture-text edge frame groups comprises: The picture-text border area is calculated by using a numpy matrix.

5. The graph matching method of claim 4, wherein, The picture-text border area is calculated by using a numpy matrix. The picture-text border area corresponding to the filtered picture-text border group is calculated by using the numpy matrix according to the following formula: S=( y n ’- y m )*( x n ’- x m ) wherein S is the picture frame area of the graphic, y m is a vertical coordinate value of the first picture frame point, y n is a vertical coordinate value of the first text frame point, x m is a horizontal coordinate value of the first picture frame point, x n is a horizontal coordinate value of the first text frame point.

6. An image-text matching apparatus for executing the image-text matching method according to any one of claims 1 to 5, wherein The picture-text border area corresponding to the filtered picture-text border group is calculated by using the numpy matrix according to the following formula: An image acquisition module is configured to acquire an initial image containing a plurality of candidate pictures and a plurality of candidate texts. A frame generation module is configured to identify the initial image, generate a picture frame for each candidate picture in the initial image, and generate a text frame for each candidate text in the initial image. A frame point selection module is configured to perform first picture frame point selection on each picture frame according to a preset first selection rule, and perform first text frame point selection on each text frame according to a preset second selection rule. A frame combination module is configured to combine the first picture frame point with each first text frame point to obtain a plurality of picture-text border groups. A frame filtering module is configured to filter picture-text border groups whose coordinate values of the first text frame points do not satisfy a preset matching condition. A text matching module is configured to calculate picture-text border areas corresponding to the filtered picture-text border groups, and select a candidate text in a picture-text border with the smallest picture-text border area as a candidate text matched with the picture frame corresponding to the first picture frame point.

7. A computer device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that, The processor executes the computer program to implement the steps of the picture-text matching method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the picture-text matching method in any one of claims 1 to 5.

Images