Image processing method, device, apparatus and storage medium

Abstract

The application provides an image processing method, device and equipment and a storage medium. The specific implementation scheme is as follows: attribute information of each graphical element in a first image is acquired, the graphical elements constitute a first graphical object in the first image, graph structure information of the first graphical object is determined according to the attribute information of each graphical element in the first image, and graph structure information of a second graphical object in a second image is acquired; and at least according to the graph structure information of the first graphical object and the graph structure information of the second graphical object, a graphical region matched with the first graphical object is determined from the second graphical object. According to the technical scheme of the application, the accuracy of puzzle image matching can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, and particularly relates to an image processing method, device and equipment and a storage medium. BACKGROUND

[0002] The puzzle pattern refers to a pattern formed by specific graphic elements according to specific position relationships, such as building block puzzles, clever board puzzles, characters and the like, which are essentially graphic objects with specific meanings formed by some standard graphic elements according to specific position relationships.

[0003] Due to the characteristics of the graphic elements of the puzzle pattern, such as various changes and complex combinations, the difficulty of image matching between puzzle images is greatly increased. The common puzzle image matching method is to extract image features of the puzzle image, and then match the two puzzle images by using the overall image features to determine whether the graphics in the two puzzle images match. When the image matching method based on image features is used in puzzle image matching, the effect is not good, and the matching graphic objects are often not accurately recognized. SUMMARY

[0004] In order to solve the above problems, the present application provides an image processing method, device, equipment and storage medium, which can significantly improve the accuracy of puzzle image matching.

[0005] According to a first aspect of an embodiment of the present application, an image processing method is provided, comprising:

[0006] Obtaining attribute information of each graphic element in a first image, the each graphic element constituting a first graphic object in the first image, the attribute information including at least one of the following: type, centroid position and rotation angle of the graphic element;

[0007] According to the attribute information of each graphic element in the first image, determining graph structure information of the first graphic object, and obtaining graph structure information of a second graphic object in a second image; the graph structure information includes at least one of the following: relative position information and connection relationship information between the graphic elements;

[0008] At least according to the graph structure information of the first graphic object and the graph structure information of the second graphic object, determining a graphic region matching the first graphic object from the second graphic object.

[0009] According to a second aspect of an embodiment of the present application, an image processing device is provided, comprising:

[0010] An acquisition module is configured to acquire attribute information of each graphical element in the first image, the graphical elements constituting a first graphical object in the first image, the attribute information including at least one of the following: type, centroid position and rotation angle of the graphical element;

[0011] A determination module is configured to determine graph structure information of the first graphical object according to the attribute information of each graphical element in the first image, and acquire graph structure information of a second graphical object in a second image; the graph structure information including at least one of the following: relative position information and connection relationship information between the graphical elements;

[0012] A processing module is configured to determine a graphical region matching the first graphical object from the second graphical object according to at least the graph structure information of the first graphical object and the graph structure information of the second graphical object

[0013] The third aspect of the present application provides an electronic device, comprising:

[0014] a memory and a processor;

[0015] The memory is connected with the processor, and is configured to store a program;

[0016] The processor, by running the program in the memory, implements the image processing method described above.

[0017] The fourth aspect of the present application provides a storage medium, the storage medium has a computer program stored thereon, and the computer program, when being run by a processor, implements the image processing method described above.

[0018] One of the above-mentioned embodiments has the following advantages or beneficial effects:

[0019] The attribute information of each graphical element in the first image is acquired, each graphical element constitutes a first graphical object in the first image, the graph structure information of the first graphical object is determined according to the attribute information of each graphical element in the first image, and the graph structure information of a second graphical object in a second image is acquired; and at least according to the graph structure information of the first graphical object and the graph structure information of the second graphical object, a graphical region matched with the first graphical object is determined from the second graphical object. As can be seen, at least one of the type, the center position and the rotation angle is used to represent the graphical element, so that the image information can be sparsified, the original image irrelevant information interference is eliminated, and the graphical element can be uniquely represented. Then, the relative position information and the connection relationship information (that is, the graph structure information) between the graphical elements are determined according to the attribute information of each graphical element, so that the relative relationship between the graphical elements can be represented, the relationship between each graphical element in the first graphical object and the second graphical object can be accurately represented, and the first graphical object and the second graphical object can be matched by using the graph structure information, so that the image matching accuracy can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0021] FIG. 1 The scene schematic diagram of the first graphical object and the second graphical object in the application scenario provided by the embodiments of the present application;

[0022] FIG. 2 The schematic diagram of highlighting the graphical region matched with the first graphical object in the second graphical object in the application scenario provided by the embodiments of the present application;

[0023] FIG. 3 The flow schematic diagram of the image processing method provided by the embodiments of the present application;

[0024] FIG. 4 The schematic diagram of extracting attribute information provided by the embodiments of the present application;

[0025] FIG. 5 The schematic diagram of graph matching provided by the embodiments of the present application;

[0026] FIG. 6 The structural schematic diagram of the image processing device provided by the embodiments of the present application;

[0027] FIG. 7A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0029] Example implementation environment

[0030] The application scenario examples of the image processing method provided in the specification can be a character combination scenario (i.e., a target text is formed according to multiple characters), a jigsaw puzzle scenario (including a building block jigsaw puzzle, a clever board game jigsaw puzzle, etc.), a character spelling scenario (i.e., a target character is spelled according to strokes), and the like.

[0031] The image processing method can be applied to a terminal device or a server. Specifically, the terminal device can be any device capable of running an application or a mini-program, such as a mobile phone, a computer, a tablet computer, a wearable device, and the like. The server can be any type of server, such as a cloud server. In this scenario, the terminal device runs a mini-program or an application, and the user interacts with the mini-program or the application on the terminal device, so that the target device can execute the image processing method.

[0032] Based on this, as shown in FIGS. 1-2 , the user opens a mini-program or an application and uploads a first image. The terminal device can directly process the first image, or the terminal device can upload the first image to a server for processing, which is not limited herein. In the processing process, the attribute information of each graphical element in the first image is obtained, and each graphical element constitutes a first graphical object in the first image. Then, the graph structure information of the first graphical object is determined according to the attribute information of each graphical element in the first image, and the graph structure information of a second graphical object in a second image is obtained. A graphical region matching the first graphical object is determined from the second graphical object. The second graphical object is displayed on the display module (such as a screen) of the terminal device, and the graphical region matching the first graphical object is highlighted in the second graphical object, for example, by displaying the graphical region matching the first graphical object in the second graphical object in a highlighted manner (the shaded part in FIG. 2 represents the highlighted display). In this way, the problem that the matching effect is poor and the matching graphical object cannot be accurately identified in the jigsaw puzzle image matching is solved, and the user can be reminded according to the matching graphical region, realizing the interaction between the real and the virtual.

[0033] Example method

[0034] In an example embodiment, an image processing method is provided, FIG. 3 is a flowchart of an image processing method according to an embodiment of the present application, the method comprising:

[0035] S110, obtaining attribute information of each graphical element in a first image, the each graphical element constituting a first graphical object in the first image, the attribute information comprising at least one of the following: type, centroid position and rotation angle of the graphical element;

[0036] S120, determining graph structure information of the first graphical object according to the attribute information of each graphical element in the first image, and obtaining graph structure information of a second graphical object in a second image; the graph structure information comprising at least one of the following: relative position information and connection relationship information between graphical elements;

[0037] S130, determining a graphical region matching the first graphical object from the second graphical object according to at least the graph structure information of the first graphical object and the graph structure information of the second graphical object.

[0038] In step S110, the first image represents an image containing at least one graphical element. The first image can be obtained by a camera or scanning. The graphical element refers to an element used to combine to constitute a graphical object, which can be specifically a point, a line, or a planar graphical element. For example, color blocks in a clever board game, words in a word puzzle, strokes in a word-building game, etc. are all graphical elements used to combine to constitute a graphical object. The graphical object constituted by the combination of color blocks, words and strokes is a graphical object constituted by graphical elements. The first graphical object represents a graph constituted by graphical elements in the first image.

[0039] The type of the graphical element can be distinguished by the outer contour of the graphical element, i.e. the shape constituted by the outer contour of the graphical element, such as a square, a rectangle, a circle, a sector, etc.

[0040] The centroid position of the graphical element represents the position of the center of gravity of the graphical element in the image, which can be specifically represented by the position coordinates of the center of gravity of the graphical element in the image. Further, the centroid position of the graphical element can also include relative position information between the centroids of different graphical elements in the first image, such as the distance, relative direction, relative position coordinates, etc. between the centroids of different graphical elements.

[0041] The rotation angle of the graphic element represents the angle difference between the state of the graphic element and the 0-degree angle state of the graphic element. The 0-degree angle state of the graphic element is a preset placement state of one graphic element. The angle difference between the graphic element in any state and the state of the graphic element in the 0-degree angle state is the rotation angle of the graphic element.

[0042] To avoid the problem of large regression loss of similar rotation angles, such as rotation angles 0° and 359° are very similar, but the regression loss of the two is large, in the embodiment, the rotation angle of the graphic element is represented as a direction vector, that is, the rotation angle is represented as In this way, the problem of large regression loss caused by similar rotation angles can be avoided.

[0043] Further, since the periodicity of different graphic elements (such as color blocks) is different, such as the rotation period of a square is 90°, and the rotation period of an equilateral triangle is 120°, in order to make the direction vector representation period uniform, the rotation angle is normalized to 360° before calculating the direction vector, for example, the rotation period of a square is 90°, so the direction vector representation of the rotation angle of a square graphic element is not but Similarly, the direction vector representation of the rotation angle of an equilateral triangle graphic element is and so on.

[0044] Specifically, taking a jigsaw puzzle game as an example, the jigsaw puzzle game includes a plurality of color blocks, which can be the same color or different colors. A plurality of color blocks of the same type or different types are combined and assembled to form a jigsaw puzzle pattern. All color blocks in the real space are photographed to obtain a first image.

[0045] Then, the attribute information of each color block in the first image is obtained. For example, the attribute information of the color block (i.e., the graphic element) in the first image can be identified by using a pre-trained graphic detection model. The pre-trained graphic detection model can be obtained by training the color block attribute information identification according to the attribute information of the color block sample.

[0046] The color blocks (i.e., graphic elements) in the first image can also be manually labeled and determined, and the types of the color blocks are determined, and the 0-degree angle state of the color blocks is determined according to the types of the color blocks, the rotation angle of the color blocks is determined by comparing the color blocks in the first image with the 0-degree angle state of the color blocks, and the direction vector is used for representation. The corresponding centroid selection method is determined according to the type of the color block, and the centroid of the color block is determined according to the corresponding centroid selection method, and then the centroid position coordinates of each color block and the relative position information between the centroids of each color block can be determined. For example, the center of the circumscribed rectangle of a square, regular hexagon or circle is the centroid position of the color block; the center of the circumscribed rectangle of a sector, semicircle or triangle is not equal to the centroid.

[0047] In step S120, the graph structure is composed of a finite non-empty set of vertices and a set of edges between the vertices, and in the embodiment of the present application, the graph structure is composed of the graphic elements as the vertices and the relationships between the graphic elements as the edges. The graph structure information contained in the graph structure represents the spatial structure relationship of each graphic element, and particularly contains the relative position information and connection relationship information between the graphic elements.

[0048] After obtaining the attribute information of each graphic element in the first image through step S110, that is, obtaining the type, centroid position and rotation angle information of each graphic element in the first image, based on these information, the graph structure of each graphic element in the first image is constructed, such as connecting each graphic element in the first image to form a connection graph of each graphic element as the graph structure. Then, for any two connected graphic elements in the graph structure, the relative position relationship between the two graphic elements is calculated according to the centroid position and the rotation angle of the graphic elements, such as the rotation relationship, the relative position coordinates, the relative distance, etc., as the connection edge features of the two graphic elements. According to the above processing, the spatial structure information between any two graphic elements in the graph structure can be determined, such as the relative position information and the connection relationship information.

[0049] The second image mentioned above refers to an image containing a graphic object composed of graphic elements. The second image can be obtained by a camera, can be scanned, or can be a computer-generated and pre-stored image. The second image is image data containing a template graphic, and the second graphic object represents a template graphic composed of graphic elements. For the second image mentioned above, the graph structure information of the second graphic object therein is also obtained.

[0050] Exemplarily, the graph structure information of the second graph object can be pre-stored, so that the graph structure information of the first graph object is read from the pre-stored position. It can also be that the attribute information of each graph element in the pre-stored second image is extracted, and the saved attribute information is used to obtain the graph structure information of the second graph object according to the method of obtaining the graph structure information of the first graph object. The relative position information between the graph elements can be calculated according to the rotation angles of the graph elements. The target graph element is selected according to the distance between the centroid positions of the graph elements, and the connection relationship information between the target graphs and the relative position information of each graph element are used as the graph structure information.

[0051] Through the processing of steps S110 and S120, the image features of the graph elements in the first image are obtained, i.e., the type, centroid position, rotation angle, etc. of the graph elements, and the relative information of different graph elements, such as the relative position information between different graph elements, the mutual connection relationship, etc. are obtained, so that the image processing method proposed in the embodiment of the present application can realize the image information sparsification processing for the jigsaw image, rather than directly extracting the information of the whole image, so as to more accurately grasp the information of the image content, i.e., to obtain the image detail information, so as to provide more abundant information reference for the image matching, which is beneficial to more accurately perform the image matching.

[0052] In step S130, exemplarily, the graph structure information of the first graph object can be compared with the graph structure information of the second graph object, and the part in the second graph object that matches the graph structure information of the first graph object is searched as the graph region matched with the first graph object. It can also be that the graph matching algorithm is used to calculate the graph structure information of the first graph object and the graph structure information of the second graph object, and determine the graph region matched with the first graph object and the second graph object.

[0053] In the technical solution of the present application, attribute information of each graphical element in the first image is obtained, each graphical element constitutes a first graphical object in the first image, graph structure information of the first graphical object is determined according to the attribute information of each graphical element in the first image, and graph structure information of a second graphical object in a second image is obtained; and at least according to the graph structure information of the first graphical object and the graph structure information of the second graphical object, a graphical region matched with the first graphical object is determined from the second graphical object. As can be seen, the graphical elements are represented by type, centroid position and rotation angle, which can sparse the image information, eliminate irrelevant information interference of the original image, and uniquely represent the graphical elements, facilitating subsequent client rendering. Then, the relative position information and connection relationship information (i.e. graph structure information) between the graphical elements determined according to the attribute information of each graphical element are used to represent the relative relationship between the graphical elements, which can accurately represent the relationship between each graphical element in the first graphical object and the second graphical object, so that the first graphical object and the second graphical object are matched by using the graph structure information, which can improve the accuracy of image matching.

[0054] In an embodiment, the attribute information of each graphical element in the first image is obtained by:

[0055] The first image is input into a pre-trained graphical detection model, so that the graphical detection model obtains the attribute information of each graphical element in the first image.

[0056] The graphical detection model is trained at least by extracting attribute information of graphical elements in a sample image, and the sample image contains a graphical object composed of graphical elements.

[0057] Exemplarily, the sample image can be an image obtained by a specified camera. Specifically, the shooting position of the specified camera is fixed, and the graphical elements are randomly placed and photographed. Then, the distortion of the sample image is corrected according to the intrinsic and extrinsic parameters of the specified camera. After obtaining the sample image, the outer contour of the graphical elements in the sample image is labeled, the centroid and rotation angle of each graphical element are calculated according to the contour, the rotation angle is all normalized to 360°, and the recognition box and the type of each graphical element are determined, so as to obtain the sample image label.

[0058] Exemplarily, the graphical detection model described above includes an image feature extraction module, a rotation angle prediction module, a type prediction module and a centroid position prediction module, and in addition, a graphical element recognition module. Each of the above modules can be implemented by any neural network capable of realizing the corresponding function, such as a backbone network for the image feature extraction module, and a convolutional neural network for other modules.

[0059] The image feature extraction module is configured to extract features of the first image. The rotation angle prediction module, the type prediction module, and the centroid position prediction module, and the graphic element identification module are configured to respectively identify a rotation angle of a graphic element, a graphic element type, a graphic element centroid position, and the graphic element in the first image based on the image features of the first image.

[0060] During training of the graphic detection model, a sample image is input to the image feature extraction module to output a sample image feature. The sample image feature is respectively input to the rotation angle prediction module, the type prediction module, and the centroid position prediction module, and the graphic element identification module for training to obtain a model output of a rotation angle, a graphic element type, a graphic element centroid position, and a graphic element bounding box.

[0061] Then, a loss function is calculated based on the model output and a sample image label corresponding to the sample image. In the embodiment of the present application, the loss function includes a first loss function corresponding to the rotation angle prediction, a second loss function corresponding to the graphic element type prediction, a third loss function corresponding to the centroid position prediction, and a fourth loss function corresponding to the graphic element identification. The various loss functions described above can use the same loss function or different loss functions, which are not limited in the embodiment of the present application. As an optional implementation, in the embodiment of the present application, the first loss function uses cosine similarity, the second loss function uses Ce-loss, the third loss function uses L1-loss, and the fourth loss function uses G-IOUloss.

[0062] In the training process, the first loss function, the second loss function, the third loss function, and the fourth loss function can be calculated according to a preset weight coefficient to obtain a total loss function. The preset weight coefficient can be set according to actual needs, which is not limited herein. The formula of the total loss function is as follows:

[0063] Loss = a1*Loss cls + a2*Loss bbox + a3*Loss ceniroid + a4*Loss angle

[0064] Wherein, a1, a2, a3, a4 represent weight coefficients, Loss cls represents the second loss function, Loss bbox represents the fourth loss function, Loss centroid represents the third loss function, and Loss angle represents the first loss function.

[0065] The trained graphic detection model is obtained by constraining and optimizing the graphic detection model by the loss function until the loss function meets the training requirements.

[0066] The first image is input into the trained graphic detection model, and attribute information of each graphic element in the first image output by the graphic detection model is obtained.

[0067] In an embodiment, the specific processing process of the image processing method or the graphic detection model for obtaining attribute information of each graphic element in the first image includes:

[0068] The first encoding feature, the second encoding feature and the third encoding feature are extracted from the first image, wherein the first encoding feature is used to identify the type of each graphic element in the first image, the second encoding feature is used to identify each graphic element in the first image, and the third encoding feature is used to identify the rotation angle of each graphic element in the first image;

[0069] The attribute information of each graphic element in the first image is determined based on the first encoding feature, the second encoding feature and the third encoding feature.

[0070] Specifically, first, the first image is feature extracted to obtain a first image feature. Since the attribute information of the graphic element to be identified is attribute information of different dimensions, and based on the same image feature, it is usually difficult to simultaneously realize attribute recognition of different dimensions. In order to facilitate the realization of attribute recognition of different dimensions, the first image feature is encoded in the embodiment of the application, so that the encoded feature can be suitable for specific attribute recognition.

[0071] Specifically, the first image feature is encoded into a first encoding feature suitable for graphic element type recognition, and based on the first encoding feature, the type of the graphic element in the first image can be identified; the first image feature is encoded into a second encoding feature suitable for identifying the graphic element in the first image, and based on the second encoding feature, the graphic element in the first image can be identified, and the centroid position of the graphic element in the first image can be identified; the first image feature is encoded into a third encoding feature suitable for identifying the rotation angle of each graphic element in the first image, and based on the third encoding feature, the rotation angle of the graphic element in the first image can be identified.

[0072] Exemplarily, see FIG. 4As shown, the backbone network of the graphic detection model extracts first image features of the first image, and respectively inputs the first image features into a classification encoder (cls-encoder), a bounding box encoder (bbox-encoder) and an angle encoder (angle-encoder) to encode the first image features according to the three encoders to obtain first encoding features, second encoding features and third encoding features. Since the center position and target detection are both realized through the bounding box, the second encoding features can output different results through different task heads, specifically, the center position and the identification of the graphic elements can be realized through the bounding box regression task head and the center task head.

[0073] The first encoding features are input into a classification task head (cls-head) to output the graphic element type; the second encoding features are respectively input into a center task head (centroid-head) and a bounding box regression task head (bbox-head) to output the center position and the graphic element identification result in the first image (that is, each graphic element in the first image is identified through the bounding box); and the third encoding features are input into an angle task head (angle-head) to output the rotation angle.

[0074] In the embodiments of the present application, different attributes of the graphic elements are identified through different encoding features, the interference of the neighborhood information and the background is eliminated, the attribute identification of the graphic elements is more accurate, and the matching result of the image is more accurate.

[0075] In some implementations, based on the first encoding features, the second encoding features and the third encoding features, attribute information of each graphic element in the first image is determined, including:

[0076] Based on the first encoding features, the type of each graphic element in the first image is determined; based on the first encoding features and the second encoding features, the center position of each graphic element in the first image is determined; and based on the first encoding features and the third encoding features, the rotation angle of each graphic element in the first image is determined.

[0077] Specifically, due to the differences in the rotation period, shape, size, etc. of different graphic elements, the difficulty of predicting the rotation angle of the graphic element is increased. In order to more accurately predict the rotation angle of the graphic element, the graphic element type information is integrated when predicting the rotation angle of the graphic element, so as to alleviate the rotation angle prediction bias caused by different processing methods for different graphic elements. That is, when determining the rotation angle of each graphic element in the first image, not only the third encoding feature is used as the basis, but also the first encoding feature is referred to, that is, the third encoding feature and the first encoding feature are fused, for example, the third encoding feature and the first encoding feature are fused by using the concat method, and then the fusion feature is used to identify the rotation angle of each graphic element in the first image. By introducing the graphic element type information, the graphic element type can be referred to when predicting the rotation angle of the graphic element, so that the influence of different graphic element types on identifying the rotation angle of the graphic element can be reduced, and the difficulty of predicting the rotation angle can be reduced.

[0078] The center position prediction is similar to the normal center point prediction, but the relationship between the center of mass of different graphic elements and the center of the graphic element circumscribed rectangle is also different, that is, the judgment method of the center of mass of different types of graphic elements is different. Therefore, when predicting the center position, there is a bias in the calculation method of different graphic elements, so the graphic element type information is also added when predicting the center position. That is, when determining the center position of each graphic element in the first image, not only the second encoding feature is used as the basis, but also the first encoding feature is referred to, that is, the second encoding feature and the first encoding feature are fused, for example, the second encoding feature and the first encoding feature are fused by using the concat method, and then the fusion feature is used to identify the center position of each graphic element in the first image. By introducing the graphic element type information, the graphic element type can be referred to when predicting the center position of the graphic element, so that the influence of different graphic element types on identifying the center position of the graphic element can be reduced, so as to improve the identification accuracy of the center position of the graphic element. Moreover, since the center position prediction is related to the graphic element circumscribed rectangle, the center position prediction can be realized by using the same model as the image element recognition.

[0079] As FIG. 4As shown, taking a puzzle game as an example, the first image is composed of a plurality of color blocks (i.e. graphic elements), the color block features are extracted by inputting the first image into a backbone feature extraction network (such as ResNet, Bert), and the color block features are input into a classification encoder, a bounding box encoder and an angle encoder respectively to obtain first encoding features, second encoding features and third encoding features. The first encoding features are input into a classification task head to output the types of each color block; the first encoding features and the second encoding features are input into a centroid task head to output the centroid positions, the second encoding features are input into a bounding box regression task head to output the bounding boxes of the color blocks in the first image; and the first encoding features and the third encoding features are input into an angle task head to output the rotation angles. In this way, the type information is added when predicting the centroid positions and the type information is added when predicting the rotation angles, so as to avoid the bias caused by different processing methods of different color block types and reduce the prediction difficulty.

[0080] In an embodiment, the graph structure information of the first graphic object is determined according to the attribute information of each graphic element in the first image, including:

[0081] According to the attribute information of each graphic element in the first image, a target graphic element is selected from each graphic element in the first image, wherein the distance between any two of the selected target graphic elements does not exceed a set distance;

[0082] According to the attribute information of each target graphic element, the graph structure information of each target graphic element is determined.

[0083] For example, the distance between graphic elements can be determined according to the distance of the centroid positions, and then the distance between the graphic elements is compared with the set distance. If the distance between two graphic elements does not exceed the set distance, the two graphic elements are taken as a group of target graphic elements. The set distance can be set according to actual needs, which is not limited herein, for example, 200 pixel values. For each group of target graphic elements, the cosine similarity of the rotation angle of each group of target graphic elements is calculated, and then the cosine similarity of the rotation angle of the target graphic elements and the relative distance of the graphic elements are taken as the relative position information of the graphic elements, and the relationship of each group of target graphic elements is taken as the connection relationship. The relative position relationship and the connection relationship of each group of target graphic elements are taken as the graph structure information of the first graphic object. In this way, the relationship between the graphic elements can more accurately represent the relationship between the elements in the image, so that the region matched by the first graphic object in the second graphic object can be more accurately determined.

[0084] Preferably, the graph structure information of each target graphic element is determined according to the attribute information of each target graphic element, including:

[0085] connecting the selected target graphic elements to obtain a graphic element connection graph;

[0086] determining relative position information between each two connected target graphic elements in the graphic element connection graph according to the attribute information of the target graphic elements;

[0087] taking the relative position information between the connected target graphic elements as a characteristic value of a connection edge between the connected target graphic elements to obtain a graphic element structure graph, the graphic element structure graph containing graph structure information of each target graphic element.

[0088] Specifically, the distance between the graphic elements is compared with a set distance, and two graphic elements not exceeding the set distance are taken as a group of target graphic elements. The centroids of the target graphic elements are connected to obtain a graphic element connection graph. The cosine similarity of the rotation angles of the two graphic elements of each connection edge in the graphic element connection graph and the relative distance are calculated to obtain the relative position information between the two graphic elements of each connection edge. The relative position information is taken as the characteristic value of the connection edge. In this way, the characteristic values of the connection edges and the connection edges (i.e., connection relationships) are combined to obtain a graphic element structure graph.

[0089] In this embodiment, taking a puzzle game as an example, after the attribute information in the first image is determined, two target color blocks whose distance between the centroid positions does not exceed a set distance are screened out, and the two target color blocks are taken as a group of target color blocks. Then the centroids of the group of target color blocks are connected to form a graphic element connection graph of the first graphic object. For the graphic element connection graph of the first graphic object, the connection relationship of each group of target color blocks is taken as a connection edge, the cosine similarity of the rotation angles of each group of target color blocks and the relative distance are calculated as the relative position information (i.e., the characteristic value of the connection edge) of each group of target color blocks. In this way, the characteristic values of the connection edges and the connection edges are combined to obtain a graphic element structure graph of the first graphic object.

[0090] In an implementation, obtaining graph structure information of a second graphic object in a second image includes:

[0091] determining the graph structure information of the second graphic object in the second image according to the attribute information of each graphic element in the second image obtained in advance, wherein each graphic element in the second image constitutes the second image object.

[0092] Specifically, taking a puzzle game as an example, the second image includes a template image composed of multiple color blocks, and the second graphic object represents a graph structure formed by connecting the centroids of the multiple color blocks.

[0093] In the embodiment, the attribute information of each color block in the second image, such as the position of the centroid, the rotation angle and the type, is calculated in advance and saved. After the graph structure information of the first graph object is obtained, the attribute information of each color block in the second image, such as the position of the centroid, the rotation angle and the type, is extracted. Then, the graph element connection graph of the graph elements in the second image is constructed according to the above embodiment, that is, two target color blocks whose distance between the positions of the centroids does not exceed the set distance are screened out, and the two target color blocks are taken as a group of target color blocks. Then, the centroids of the group of target color blocks are connected to form the graph element connection graph of the second graph object.

[0094] For the graph element connection graph of the second graph object, the cosine similarity of the rotation angle of each group of target color blocks and the relative distance are calculated as the relative position information, and the relationship of each group of target color blocks is taken as the connection relationship information. In this way, the attribute information of each graph element in the second image is saved in advance, which can improve the matching time of the image, reduce the calculation amount and improve the matching efficiency.

[0095] In an embodiment, the graph region matched with the first graph object is determined from the second graph object according to at least the graph structure information of the first graph object and the graph structure information of the second graph object, step S140, including:

[0096] According to the attribute information of each graph element in the first image, the attribute information of each graph element in the second image, the graph structure information of the first graph object and the graph structure information of the second graph object, the graph region matched with the first graph object is determined from the second graph object; wherein each graph element in the second image constitutes the second graph object in the second image.

[0097] Exemplarily, the type of the graph element is encoded into a vector, the rotation angle is encoded into a direction vector, and the position of the centroid is adjusted according to the specified size. The encoded type, rotation angle and position of the centroid are spliced to obtain the node feature.

[0098] Specifically, taking the jigsaw game as an example, after obtaining the type, centroid position and rotation angle of each color block, the type of each color block is encoded into a one-hot vector, the rotation angle of each color block is encoded into a direction vector, and the centroid position of each color block is normalized according to the image size, that is, the centroid position is adjusted to between 0 and 1. The encoded type, rotation angle and centroid position of each color block are spliced to obtain the node features of each color block. It can be understood that the attribute information of each color block in the first image and the attribute information of each color block in the second image are processed respectively by the above-mentioned manner, and the first node features of each color block in the first image and the second node features of each color block in the second image are obtained respectively.

[0099] The graph structure information is taken as the edge features of the graph objects, that is, the graph structure information of the first graph object is taken as the first edge feature, and the graph structure information of the second graph object is taken as the second edge feature. As shown in FIG. 5 The first node features of each color block in the first image, the second node features of each color block in the first image, the first edge features of the first graph object and the second edge features of the second graph object are calculated by using a graph matching algorithm to determine the graph region of the first graph object in the second graph object. The graph matching algorithm can be a random walk-based matching algorithm, and other graph matching algorithms can also be used, which are not limited herein. In this way, the color blocks in the real space are encoded into discrete and sparse color block information, eliminating the interference of irrelevant information of the original image. Based on the encoded attribute information, the discrete color blocks are connected by a graph, and the node features, edge features and other features are calculated to represent the absolute and relative relationships between the color blocks. Finally, the matching relationship between the color blocks is determined by the graph matching algorithm.

[0100] Example apparatus

[0101] Correspondingly, FIG. 6 is a structural schematic diagram of an image processing device according to an embodiment of the present application. In an exemplary embodiment, an image processing device is provided, comprising:

[0102] The acquisition module 610 is configured to acquire attribute information of each graph element in a first image, wherein the graph elements constitute a first graph object in the first image, and the attribute information includes at least one of the following: a type, a centroid position and a rotation angle of the graph element.

[0103] The determination module 620 is configured to determine graph structure information of the first graph object according to the attribute information of each graph element in the first image, and acquire graph structure information of a second graph object in a second image; the graph structure information includes at least one of the following: relative position information and connection relationship information between graph elements.

[0104] The processing module 630 is configured to determine a graphic region matched with the first graphic object from the second graphic object according to the graph structure information of the first graphic object and the graph structure information of the second graphic object.

[0105] In an embodiment, the obtaining module 610 comprises:

[0106] The attribute determining module is configured to input the first image into a pre-trained graphic detection model, so that the graphic detection model obtains attribute information of each graphic element in the first image.

[0107] The graphic detection model is trained by at least extracting attribute information of graphic elements in a sample image, and the sample image comprises a graphic object composed of graphic elements.

[0108] In an embodiment, the obtaining module 610 comprises:

[0109] The first encoding module is configured to extract a first encoding feature, a second encoding feature and a third encoding feature from the first image, wherein the first encoding feature is used to identify a type of each graphic element in the first image, the second encoding feature is used to identify each graphic element in the first image, and the third encoding feature is used to identify a rotation angle of each graphic element in the first image.

[0110] The second encoding module is configured to determine attribute information of each graphic element in the first image based on the first encoding feature, the second encoding feature and the third encoding feature.

[0111] In an embodiment, the second encoding module is further configured to:

[0112] determine a type of each graphic element in the first image based on the first encoding feature, determine a centroid position of each graphic element in the first image based on the first encoding feature and the second encoding feature, and determine a rotation angle of each graphic element in the first image based on the first encoding feature and the third encoding feature.

[0113] In an embodiment, the determining module is further configured to:

[0114] select a target graphic element from each graphic element in the first image according to the attribute information of each graphic element in the first image, wherein a distance between any two of the selected target graphic elements does not exceed a set distance.

[0115] determine graph structure information of each target graphic element according to the attribute information of each selected target graphic element.

[0116] In an embodiment, the graph structure information of each target graph element is determined according to the attribute information of each selected target graph element, including:

[0117] The selected target graph elements are connected to obtain a graph element connection graph.

[0118] According to the attribute information of each target graph element, the relative position information between each two connected target graph elements in the graph element connection graph is determined.

[0119] The relative position information between the connected target graph elements is taken as the characteristic value of the connection edge between the connected target graph elements to obtain a graph element structure graph, and the graph element structure graph contains the graph structure information of each target graph element.

[0120] In an embodiment, the graph structure information of a second graph object in a second image is obtained, including:

[0121] According to the attribute information of each graph element in the second image, the graph structure information of the second graph object in the second image is determined, wherein each graph element in the second image constitutes the second graph object in the second image.

[0122] In an embodiment, at least according to the graph structure information of the first graph object and the graph structure information of the second graph object, a graph region matching the first graph object is determined from the second graph object, including:

[0123] According to the attribute information of each graph element in the first image, the attribute information of each graph element in the second image, the graph structure information of the first graph object, and the graph structure information of the second graph object, a graph region matching the first graph object is determined from the second graph object.

[0124] Wherein each graph element in the second image constitutes the second graph object in the second image.

[0125] The image processing apparatus provided by the embodiment belongs to the same application concept as the image processing method provided by the above embodiments of the present application, can execute the image processing method provided by any of the above embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the image processing method. Technical details not described in detail in the embodiment can be referred to the specific processing content of the image processing method provided by the above embodiments of the present application, which will not be described here.

[0126] Example electronic device

[0127] Another embodiment of the present application also provides an electronic device, as shown inFIG. 7 As shown, the device comprises:

[0128] The memory 700 and the processor 710;

[0129] The memory 700 is connected with the processor 710, and is configured to store programs.

[0130] The processor 710 is configured to realize the image processing method disclosed in any of the above embodiments by running the programs stored in the memory 700.

[0131] Specifically, the electronic device can further comprise a bus, a communication interface 720, an input device 730 and an output device 740.

[0132] The processor 710, the memory 700, the communication interface 720, the input device 730 and the output device 740 are connected with each other through the bus.

[0133] The bus can include a path for transmitting information between various components of the computer system.

[0134] The processor 710 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-use programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0135] The processor 710 can include a main processor, and can further include a baseband chip, a modem, etc.

[0136] The memory 700 stores programs for executing the technical solutions of the present application, and can also store an operating system and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 700 can include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash, etc.

[0137] The input device 730 can include devices that receive data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer or a gravity sensor, etc.

[0138] The output device 740 can include a device that allows output of information to the user, such as a display screen, a printer, a speaker, etc.

[0139] The communication interface 720 can include a device using any transceiver, etc., to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0140] The processor 710 executes programs stored in the memory 700 and invokes other devices, which can be used to implement each step of any image processing method provided by the embodiments described above.

[0141] Example computer program product and storage medium

[0142] In addition to the above method and device, the embodiments of the present application can also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the image processing method according to various embodiments of the present application described in the above "Exemplary Method" section of the specification.

[0143] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language, such as Java, C++, etc., and a conventional procedural programming language, such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0144] In addition, the embodiments of the present application can also be a storage medium, which stores a computer program, and the computer program is executed by a processor to perform the steps of the image processing method according to various embodiments of the present application described in the above "Exemplary Method" section of the specification, the specific working content of the above-mentioned electronic device, and the specific working content of the above-mentioned computer program product and computer program stored on the storage medium when executed by the processor, which can be referred to the content of the above-mentioned method embodiments, which will not be repeated here.

[0145] For each method embodiment described above, for the sake of simplicity, the method embodiments are described as a series of acts. But those skilled in the art will appreciate that the method embodiments are not limited by the order of acts, as some steps could occur in other orders or concurrently with each other. Moreover, those skilled in the art will appreciate that described acts could be implemented other ways, such as at least partially in hardware, and that the disclosure is not limited to the described or disclosed acts or to the order of such acts.

[0146] It should be noted that each of the above-described examples of the present disclosure are described in a progressive manner, and each example focuses on the differences from other examples. Therefore, the same or similar parts among the examples can be mutually referred to. For the device examples, since they are basically similar to the method examples, the description is relatively simple, and the relevant parts can be referred to the description of the method examples.

[0147] The steps in the method embodiments of the present disclosure can be adjusted in order, combined, and reduced according to actual needs. The technical features recorded in each embodiment can be replaced or combined.

[0148] The modules and sub-modules in the device and terminal of each embodiment of the present disclosure can be combined, divided, and reduced according to actual needs.

[0149] In several embodiments provided by the present disclosure, it should be understood that the disclosed terminal, device and method can be implemented by other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or sub-modules is only a logical function division. In actual implementation, another division mode can be used, for example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0150] The modules or sub-modules described as separate components can or can not be physically separate, and the components of the modules or sub-modules can or can not be physical modules or sub-modules, i.e. they can be located in one place or distributed on multiple network modules or sub-modules. Some or all modules or sub-modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0151] In addition, each functional module or sub-module in each embodiment of the present application can be integrated in one processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The integrated module or sub-module can be realized in the form of hardware or in the form of a software functional module or sub-module.

[0152] The skilled person can further realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, each example has been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0153] The steps of the method or algorithm described in connection with the embodiments disclosed herein can be directly implemented by hardware, a software unit executed by a processor, or a combination of both. The software unit can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0154] Finally, it should be noted that, in this document, relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0155] The above description of disclosed embodiments enables one of ordinary skill in the art to make and use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Obtain the attribute information of each graphic element in the first image, wherein each graphic element constitutes a first graphic object in the first image, and the attribute information includes: the type, centroid position, and rotation angle of the graphic element; Based on the attribute information of each graphic element in the first image, the graph structure information of the first graphic object is determined, and the graph structure information of the second graphic object in the second image is obtained; the graph structure information includes: relative position information and connection relationship information between graphic elements; Based at least on the graph structure information of the first graphic object and the graph structure information of the second graphic object, a graphic region matching the first graphic object is determined from the second graphic object; Specifically, determining the graph structure information of the first graphic object based on the attribute information of each graphic element in the first image includes: Based on the attribute information of each graphic element in the first image, target graphic elements are selected from each graphic element in the first image, wherein the distance between any two selected target graphic elements does not exceed a set distance. Connect the selected target graphic elements to obtain a graphic element connection diagram; Based on the attribute information of each target graphic element, determine the relative position information between every two connected target graphic elements in the graphic element connection diagram; Using the relative position information between connected target graphic elements as the feature values ​​of the connecting edges between connected target graphic elements, a graphic element structure diagram is obtained, which contains the graph structure information of each target graphic element.

2. The method according to claim 1, characterized in that, The step of obtaining the attribute information of each graphic element in the first image includes: The first image is input into a pre-trained image detection model so that the image detection model can obtain the attribute information of each graphic element in the first image. The image detection model is trained by extracting attribute information of image elements in a sample image, wherein the sample image contains image objects composed of image elements.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the attribute information of each graphic element in the first image includes: A first coding feature, a second coding feature, and a third coding feature are extracted from the first image. The first coding feature is used to identify the type of each graphic element in the first image, the second coding feature is used to identify each graphic element in the first image, and the third coding feature is used to identify the rotation angle of each graphic element in the first image. Based on the first encoding feature, the second encoding feature, and the third encoding feature, the attribute information of each graphic element in the first image is determined.

4. The method according to claim 3, characterized in that, The step of determining the attribute information of each graphic element in the first image based on the first coding feature, the second coding feature, and the third coding feature includes: Based on the first encoding feature, the type of each graphic element in the first image is determined; based on the first encoding feature and the second encoding feature, the centroid position of each graphic element in the first image is determined; based on the first encoding feature and the third encoding feature, the rotation angle of each graphic element in the first image is determined.

5. The method according to claim 1, characterized in that, Obtain the graph structure information of the second graphic object in the second image, including: Based on the attribute information of each graphic element in the pre-acquired second image, the graph structure information of the second graphic object in the second image is determined, wherein each graphic element in the second image constitutes the second image object.

6. The method according to claim 1, characterized in that, Based at least on the graph structure information of the first graphic object and the graph structure information of the second graphic object, a graphic region matching the first graphic object is determined from the second graphic object, including: Based on the attribute information of each graphic element in the first image, the attribute information of each graphic element in the second image, the graph structure information of the first graphic object, and the graph structure information of the second graphic object, a graphic region matching the first graphic object is determined from the second graphic object. In this context, each graphic element in the second image constitutes the second graphic object in the second image.

7. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire attribute information of each graphic element in the first image, wherein each graphic element constitutes a first graphic object in the first image, and the attribute information includes: the type of graphic element, the centroid position, and the rotation angle. The determining module is used to determine the graph structure information of the first graphic object based on the attribute information of each graphic element in the first image, and to obtain the graph structure information of the second graphic object in the second image; the graph structure information includes: relative position information and connection relationship information between graphic elements; The processing module is configured to determine, based at least on the graph structure information of the first graphic object and the graph structure information of the second graphic object, a graphic region that matches the first graphic object from the second graphic object; Specifically, determining the graph structure information of the first graphic object based on the attribute information of each graphic element in the first image includes: Based on the attribute information of each graphic element in the first image, target graphic elements are selected from each graphic element in the first image, wherein the distance between any two selected target graphic elements does not exceed a set distance. Connect the selected target graphic elements to obtain a graphic element connection diagram; Based on the attribute information of each target graphic element, determine the relative position information between every two connected target graphic elements in the graphic element connection diagram; Using the relative position information between connected target graphic elements as the feature values ​​of the connecting edges between connected target graphic elements, a graphic element structure diagram is obtained, which contains the graph structure information of each target graphic element.

8. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor implements the image processing method as described in any one of claims 1 to 6 by running the program in the memory.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 1 to 6.

Images