Image processing method, device, equipment, medium and product

Abstract

The embodiment of the present invention provides an image processing method that can be applied to the field of artificial intelligence technology. The image processing method can be applied to object placement, specifically including: obtaining target state information of a target placement state image; generating a placement state data set matching the target state information through a placement evaluation model; wherein the placement evaluation model is a neural network model that estimates the target state information based on initial state information, and the initial state information is the initial placement state information of the target placement space; and generating a placement process data set based on the placement state data set, and the placement process data set is applied to object placement. The embodiment of the present invention also provides an image processing device, equipment, storage medium and program product applied to object placement.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to the field of image processing technology, and more specifically to an image processing method, device, equipment, medium and product. Background Art

[0002] In modern industry and manufacturing, the automated placement of objects has become an important part of improving production efficiency and product quality. Object placement technology has been widely used in packaging, assembly, warehousing and logistics. Existing object placement technologies mainly rely on several methods such as robotic arm placement, visual guidance placement and template matching placement. Among them, in the process of robotic arm placement, the robotic arm grabs and places objects through preset paths and programs. This method has high accuracy and repeatability, but when facing objects of different shapes and sizes, or encountering different scene conditions, it often requires complex programming and adjustment, and has poor flexibility; in addition, visual guidance placement uses computer vision technology to capture the image of the object through the camera, identify the shape, position and orientation of the object, and then place it in combination with the algorithm. Although this method improves flexibility to a certain extent, it has high requirements for image processing and recognition algorithms, and is easily affected by factors such as light and occlusion in complex environments; template matching placement requires matching and placing objects through pre-defined templates. It works better when processing objects with regular shapes, but for objects with complex shapes and various changes, the design and matching process of the template is relatively cumbersome and difficult to adapt to diverse needs. Summary of the invention

[0003] In view of at least one of the technical problems existing in the above-mentioned prior art, the embodiments of the present invention provide an image processing method, device, equipment, medium and product that can generalize the placement of objects in various application scenarios and ensure higher precision placement of diversified objects.

[0004] One aspect of an embodiment of the present invention provides an image processing method for object placement, comprising: obtaining target state information of a target placement state image; generating a placement state data set matching the target state information through a placement evaluation model; wherein the placement evaluation model is a neural network model for evaluating the target state information based on initial state information, and the initial state information is initial placement state information of the target placement space; and generating a placement process data set based on the placement state data set, and the placement process data set is applied to object placement.

[0005] According to an embodiment of the present invention, acquiring target state information of a target placement state image includes: extracting feature vector data of the target placement state image as the target state information.

[0006] According to an embodiment of the present invention, before obtaining the target state information of the target placement state image, the method includes: obtaining a feature vector data set of a training image set; and generating a placement evaluation model according to the feature vector data set.

[0007] According to one embodiment of the present invention, obtaining a feature vector data set of a training image set includes: extracting vectorized data of each image in the training image set; and performing classification on each image in the training image set according to the vectorized data to generate a corresponding feature vector data set.

[0008] According to one embodiment of the present invention, in generating a placement evaluation model according to a feature vector data set, it includes: generating a first evaluation model based on a first neural network according to the feature vector data set; generating a second evaluation model based on a second neural network according to the feature vector data set; wherein the placement evaluation model includes the first evaluation model and the second evaluation model.

[0009] According to one embodiment of the present invention, in generating a placement state data set that matches target state information through a placement evaluation model, it includes: generating a placement state combination set that matches the target state information according to initial state information through a first evaluation model of the placement evaluation model; and performing valuation on each placement state combination of the placement state combination set through a second evaluation model of the placement evaluation model to generate a placement state data set.

[0010] According to an embodiment of the present invention, in generating a placement process data set according to a placement state data set, it includes: performing sorting according to the estimated value of each placement state data in the placement state data set to generate the placement process data set.

[0011] Another aspect of an embodiment of the present invention provides an image processing device for object placement, which includes an information acquisition module, an information evaluation module and a data generation module. The information acquisition module is used to obtain target state information of the target placement state image; the information evaluation module is used to generate a placement state data set matching the target state information through a placement evaluation model; wherein the placement evaluation model is a neural network model that estimates the target state information based on initial state information, and the initial state information is the initial placement state information of the target placement space; the data generation module is used to generate a placement process data set based on the placement state data set, and the placement process data set is applied to object placement.

[0012] Another aspect of an embodiment of the present invention provides an electronic device, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned image processing method applied to object placement.

[0013] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned image processing method applied to object placement.

[0014] Another aspect of an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned image processing method applied to object placement.

[0015] The image processing method for object placement provided by the embodiment of the present invention can at least partially solve the technical problems existing in the related art, and thus can achieve at least one of the following technical effects:

[0016] The image processing method for object placement provided by the embodiment of the present invention generates a placement state data set that matches the target state information through a placement evaluation model, and can realize valuation of each placement state data, so as to evaluate the object placement effect according to the valuation, and realize effective screening of the placement effect, thereby significantly improving the flexibility of placement in the object placement process, preventing the influence of complex environment, and greatly reducing the placement design and matching costs. At the same time, it also avoids the situation in which the objective function (such as reward function) in the prior art is difficult to establish, and can generalize the object placement of various application scenarios. At the same time, it can meet the placement requirements of irregular and diverse placement shapes (the placement shapes randomly hand-drawn by the user can also achieve accurate and efficient placement effects), and can ensure the placement of diversified objects with higher precision, which substantially improves the placement efficiency and intelligence level of the placement process.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0019] Figure 1 A diagram schematically shows an application scenario of an image processing method, apparatus, device, medium, and program product applied to object placement according to an embodiment of the present invention;

[0020] Figure 2 A flowchart of an image processing method applied to object placement according to an embodiment of the present invention is schematically shown;

[0021] Figure 3 A model generating scene graph of an image processing method applied to object placement according to an embodiment of the present invention is schematically shown;

[0022] Figure 4 A specific application scenario diagram of the image processing method applied to object placement according to an embodiment of the present invention is schematically shown;

[0023] Figure 5 A schematic structural block diagram of an image processing device applied to object placement according to an embodiment of the present invention is shown; and

[0024] Figure 6 A block diagram of an electronic device suitable for implementing an image processing method applied to object placement according to an embodiment of the present invention is schematically shown.

[0025] The above-mentioned drawings are part of the specification of the embodiments of the present invention, which illustrate exemplary embodiments of the present invention. The attached drawings and the description of the specification are used together to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description of the drawings and the following specific implementations are only exemplary and illustrative, and they cannot limit the scope of the present invention. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0027] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0028] The terms “first”, “second”, etc. used in the present invention do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0029] The directional terms used in the present invention, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0030] The words “include,” “including,” “have,” “contain,” etc. used in the present invention are open-ended terms, meaning including but not limited to.

[0031] The term "and / or" used in the present invention includes any or all combinations of the items mentioned.

[0032] Regarding the present invention, "plurality" includes "two" and "more than two"; regarding the present invention, "plurality of groups" includes "two groups" and "more than two groups".

[0033] The terms "substantially" and "approximately" used in the present invention are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0034] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0036] In the process of global economic growth and market globalization, the demand for intelligent placement of objects in modern industry and life is increasing. Intelligent placement technology involves a variety of intelligent technologies such as artificial intelligence, big data, and the Internet of Things. It can mainly provide high space utilization, reduce costs, improve efficiency and accuracy in various industrial application scenarios such as warehousing, packaging, assembly, and logistics. It can also enhance the life experience for users in daily life. Traditional placement technology is usually based on manual decision-making, and it depends on the experience of personnel to determine the placement method. Later, it further developed into a variety of intelligent and automated placement methods such as robotic arm placement, visual guidance placement, and template matching placement. Although these methods can significantly improve the existing placement efficiency, they still have poor flexibility, are easily affected by complex environments, and have high placement design and matching costs. The level of intelligence is still low and needs to be further improved.

[0037] Furthermore, existing placement methods such as robotic arm placement, visual guidance placement, and template matching placement also need to use neural networks for machine learning to improve the level of intelligent operation, but such solutions still have certain problems. First, the objective function is difficult to set up, because existing technologies mainly rely on reinforcement learning based on reward functions to complete, but the goal of object placement is not easy to set up, and the reward function is not easy to obtain; in addition, the flexibility is insufficient, especially when the scene changes or the size of the object changes, it is difficult for the robot to perfectly migrate the preset algorithm.

[0038] In summary, the placement solutions available in the prior art still have a low level of intelligence. For objects with regular shapes and placement spaces, they may be able to achieve good placement accuracy and efficiency. However, for placement requirements of irregular and diverse shapes (such as target placement shapes hand-drawn by users), especially when it involves environmental changes, it is difficult to guarantee placement efficiency, let alone placement accuracy.

[0039] In view of at least one of the above problems, embodiments of the present invention aim to provide an image processing method, apparatus, device, medium and product for object placement, thereby providing a technical solution that can generalize the placement of objects in various application scenarios and ensure the placement of diversified objects with higher precision.

[0040] One aspect of an embodiment of the present invention provides an image processing method applied to object placement, which includes: obtaining target state information of a target placement state image; generating a placement state data set matching the target state information through a placement evaluation model; wherein the placement evaluation model is a neural network model that estimates the target state information based on initial state information, and the initial state information is initial placement state information of the target placement space; and generating a placement process data set based on the placement state data set, and the placement process data set is applied to object placement.

[0041] Figure 1 An application scenario diagram schematically illustrates an image processing method, apparatus, device, medium, and program product for object placement according to an embodiment of the present invention.

[0042] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0044] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0045] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0046] It should be noted that the image processing method for object placement provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the image processing device for object placement provided in the embodiment of the present invention can generally be set in the server 105. The image processing method for object placement provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the image processing device for object placement provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0048] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The image processing method applied to object placement in the disclosed embodiment is described in detail.

[0049] One aspect of an embodiment of the present invention provides an image processing method for object placement, which includes operations S201 to S203.

[0050] In operation S201, target state information of a target placement state image is obtained;

[0051] In operation S202, a placement state data set matching the target state information is generated by a placement evaluation model; wherein the placement evaluation model is a neural network model that estimates the target state information according to the initial state information, and the initial state information is the initial placement state information of the target placement space; and

[0052] In operation S203, a placement process data set is generated according to the placement state data set.

[0053] The target placement state image is a target image that meets the placement requirements, in which the placement shape required by the placement requirements is defined. For example, in an indoor placement space, the pattern shape defined in a bird's-eye view can be used as the placement shape of the target placement object in the indoor placement space when performing the placement operation. For example, if the target placement object is 6 chairs that are randomly placed in the indoor placement space, and the indoor space is an area of ​​10×10m 2 If the target image of the target placement state image is defined as a straight line, the existing 6 chairs should eventually be placed in a straight line to match the target placement state image.

[0054] The target placement state image may be an irregular shape image hand-drawn by a user, such as Figure 3 The circle shown (approximately an "O" shape) can be drawn with different colors. Different colors can represent different objects, or different colors can represent the same object. In other words, the objects are finally distinguished by color to achieve an approximately circular arrangement.

[0055] The target state information is the expected target placement state information of the target placement space after the placement is completed, which is mainly used to define the final placement image shape, placement position (such as coordinates), color, orientation, object type, quantity and even the distance between each placement object of the target placement object in the target placement space. Correspondingly, the initial state information is the initial placement state information of the target placement space before the placement operation is performed, which is mainly used to define the initial placement image shape, placement position, color, orientation, type, quantity and mutual spacing of the target placement objects in the target placement space.

[0056] The placement evaluation model can be a generative model based on a graph neural network and corresponding evaluation functions, and a neural network model generated by combining operator learning, so as to be able to estimate the target placement requirements defined by the target state information based on the initial state information, and then generate a placement state data set based on the estimation.

[0057] The placement state data set is a data set of various placement state combinations and their corresponding valuations formed after the placement evaluation model is evaluated. For example, 6 chairs (such as chairs AF) are placed in a straight line as the target placement pattern in the target placement space. Referring to the initial placement shape or placement state of chairs AF, if the final placement is a straight line, there can be multiple placement combinations, such as ABCDEF, AFBCDE, ECFDBA, etc. Each placement combination can be evaluated by the placement evaluation model, and this evaluation is used to determine the degree of closeness between its final placement shape and the placement shape of the target placement state image defined by the target state information, such as the initial placement image shape, placement position, color, orientation, type, quantity, and mutual spacing.

[0058] The placement process data set is a "placement combination-valuation" data set after the placement combinations existing in the placement status data set are processed according to the valuation. The processing process can be to sort the placement combinations according to the valuation, thereby generating a placement combination data set arranged in order. In addition, another content of the placement process data set is a reference data set for the placement trajectory in the subsequent execution of the placement process, that is, the placement execution operation process. The placement combinations in the placement process data set can be placed in sequence from small to large according to the valuation, so that the user can intuitively feel the process of the placement process from disorder to the final placement of the target shape, thereby forming a placement trajectory (i.e., trajectory), which can provide users with an intuitive placement dynamic demonstration effect in the placement process or the placement demonstration process, so as to adapt to various placement needs.

[0059] The image processing method for object placement provided by the embodiment of the present invention generates a placement state data set that matches the target state information through a placement evaluation model, and can realize valuation of each placement state data, so as to evaluate the object placement effect according to the valuation, and realize effective screening of the placement effect, thereby significantly improving the flexibility of placement in the object placement process, preventing the influence of complex environment, and greatly reducing the placement design and matching costs. At the same time, it also avoids the situation in which the objective function (such as reward function) in the prior art is difficult to establish, and can generalize the object placement of various application scenarios. At the same time, it can meet the placement requirements of irregular and diverse placement shapes (the placement shapes randomly hand-drawn by the user can also achieve accurate and efficient placement effects), and can ensure the placement of diversified objects with higher precision, which substantially improves the placement efficiency and intelligence level of the placement process.

[0060] like Figure 2-Figure 4 As shown, according to an embodiment of the present invention, in operation S201, obtaining target state information of the target placement state image includes: extracting feature vector data of the target placement state image as the target state information.

[0061] The manifestation of the target state information can be defined by the feature vector data of the target placement state image. The feature vector data of the image is a vector representation that combines multiple features in the image through feature vectors. It can be used to define the global or local features of the image for tasks such as image recognition.

[0062] In the embodiment of the present invention, the feature vector data is mainly used to define information such as the shape, placement position, color, orientation, type, quantity, and mutual spacing of the target placement image. The extraction of the feature vector data can be achieved through the above-mentioned placement evaluation model, or through methods such as latent space, which will not be described in detail.

[0063] By extracting the feature vector data, the target placement state can be defined more accurately, thereby ensuring the accuracy of the subsequent placement process.

[0064] like Figure 2-Figure 4 As shown, according to an embodiment of the present invention, before obtaining the target state information of the target placement state image in operation S201, it includes: obtaining a feature vector data set of a training image set; and generating a placement evaluation model according to the feature vector data set.

[0065] As a neural network model generated based on graph neural network, the placement evaluation model requires data training generation in the early stage. The training image set as the training data set of the placement evaluation model mainly includes various types of image training data. For example, various types of images can include images of various shapes such as triangles, squares, straight lines, circles, etc., and can also include images of various irregular shapes, which can be used as training data for the placement evaluation model. Specifically, such image training data can be the discretized data of the corresponding training image and the corresponding image data (such as images in png format, etc.), and the corresponding training image can be discretized by a discretization algorithm.

[0066] The feature vector data set is a feature vector data set formed by performing feature vector extraction on each training image in the training image set. The feature vector data set can realize a comprehensive and accurate training process of the placement evaluation model, thereby ensuring that the placement evaluation model can achieve a more accurate and efficient evaluation effect.

[0067] like Figure 2-Figure 4 As shown, according to one embodiment of the present invention, in obtaining a feature vector data set of a training image set, it includes: extracting vectorized data of each image in the training image set; performing classification on each image in the training image set according to the vectorized data to generate a corresponding feature vector data set.

[0068] To obtain the feature vector data set, it is necessary to train a neural network in advance, compress the training images in the above training image set into the latent space (Latent Space) corresponding to the neural network, and extract the vectorized data (Embedding) of these images according to the attribute labels (such as shape labels, etc.) of these images through the latent space. Furthermore, the vectorized data can be used to classify the training images corresponding to the same or similar features (such as image shape) into one category (i.e., classification), and then calculate the corresponding feature vector (Probability Embedding Vector) for each category to form a feature vector data set. Specifically, the corresponding feature vector data set can be generated based on the vectorized data of each training image and the corresponding preset classification label.

[0069] The training image set may come from the reuse of existing training data or correspondingly labeled training data, etc., which can ensure the diversity of training images and will not be described in detail.

[0070] This can further ensure the accuracy of the basic training data of the placement evaluation model.

[0071] like Figure 2-Figure 4 As shown, according to one embodiment of the present invention, in generating a placement evaluation model according to a feature vector data set, it includes: generating a first evaluation model based on a first neural network according to the feature vector data set; generating a second evaluation model based on a second neural network according to the feature vector data set; wherein the placement evaluation model includes the first evaluation model and the second evaluation model.

[0072] The training of the placement evaluation model needs to be based on operator learning, graph neural network, and score-based generative model. Among them, the first evaluation model is a new neural network architecture based on operator learning, graph neural network, and score-based generative model, which meets the following requirements:

[0073]

[0074] It can be recorded as Score net, that is, the first evaluation model S, where x is the discretized array of the above training image set and the graph corresponding to its spatial attributes (such as orientation, color, etc.), t is the time corresponding to the model diffusion stage, and u is the characteristic vector (Probability Embedding Vector) corresponding to the distribution of the input training image. Score net can be used to define the gradient of the corresponding training image generated by adding noise at time t.

[0075] like Figure 3 As shown in the figure, the leftmost is the input training image, which represents the possible shape to be placed, which is a circle (i.e., O shape). Discrete Optimal Transport is performed on the input training image, which discretizes the training image into a corresponding array using a discretization algorithm. At the same time, the training image is input into a trained neural network (such as a VAE model Variational Autoencoder) to generate a corresponding feature vector u (Probability EmbeddingVector). Afterwards, the feature vector u and the discretized array are used together to train the Score net.

[0076] Similarly, the second evaluation model can also be a new neural network architecture based on operator learning, graph neural network and score-based generation model, which satisfies:

[0077]

[0078] It can be recorded as Energy net, that is, the second evaluation model E. Where x is the discretized array of the above training image set and the graph corresponding to its spatial attributes (such as orientation, color, etc.), t is the time corresponding to the model diffusion stage, and u is the eigenvector of the distribution corresponding to the input training image.

[0079] In addition, the relationship between Energy net and Score net is:

[0080]

[0081] in represents the coordinates of the graph node, and S represents the gradient of the graph node coordinates. In the embodiment of the present invention, the graph node coordinates can be used as the position of the target placement object in the target placement space.

[0082] Furthermore, according to the definition of Score Function, E on the right side of the above equation represents the log probability density function (logpdf) of the training image dataset at time t in the diffusion process. If a dataset is a discretized data of a certain shape, the logpdf value corresponding to the area in its dataset will be large. When the input image is similar to the placement shape, the corresponding logpdf is large, and the value function for evaluating the placement of the target object shape is very high at these input values, so Energy net can be used as the value function of the placement of the object of this shape to achieve valuation.

[0083] If the valuation range of Energy net is defined between 0 and 1 for the convenience of subsequent applications, the second evaluation model E can satisfy:

[0084]

[0085] Among them, y is the E value of the discretized state that you want to calculate at present, and α is an empirical value that can make the E value smoother. In addition:

[0086]

[0087] Among them, x iare some estimated values, usually taking the discretized data set corresponding to the target placement shape. Therefore, a scoring function (i.e., valuation model) with a value range between 0 and 1 can be generated, which can measure the quality of object placement according to a certain shape and screen the various object placements generated by the model. Among them, the 0-1 scoring value can also be fine-tuned to a 0-100 scoring threshold, which can be adjusted according to actual needs.

[0088] Therefore, the second evaluation model Energy net can realize the valuation of multiple placement combinations corresponding to the target placement state image in the inference stage (Inference) or in the application scenario of the model. If there are multiple target placement state images, a batch valuation process can be performed, so that dynamic display based on valuation changes can be further performed during the object placement process.

[0089] like Figure 4 As shown, when the evaluation is performed according to the placement evaluation model, for an input white background with a specific shape (such as Figure 4 The target placement state image of the object (the “ε” shape shown in the figure) can be generated through the placement evaluation model. Discrete placement combination states and their corresponding valuations that are similar to the target placement shape but different in style can be generated, and the placement trajectory from the initial placement state to the target placement state can be given accordingly, thereby satisfying the dynamic display during the placement execution process.

[0090] Therefore, the above-mentioned image processing method of the embodiment of the present invention can be combined with the generative model based on graph neural network and score and operator learning, etc., to achieve deep learning and generalization of the placement of objects of various shapes. Even hand-drawn irregular shapes can be accurately placed according to the valuation, and the placement is more intelligent. In addition, in this process, a valuation function E of the object placement process can be generated to evaluate the object placement effect and filter the generated placement targets, so that the evaluation output results are more accurate and efficient, while further improving the diversity of placement scenes.

[0091] like Figure 2-Figure 4 As shown, according to one embodiment of the present invention, in operation S202, a placement state data set matching the target state information is generated by a placement evaluation model, including: generating a placement state combination set matching the target state information according to initial state information by a first evaluation model of the placement evaluation model; performing valuation on each placement state combination of the placement state combination set by a second evaluation model of the placement evaluation model to generate a placement state data set.

[0092] like Figure 4As shown, the "ε" shape drawn in the lower left is a target placement state image hand-drawn by the user. The placement evaluation model of the present invention can extract its corresponding feature vector u (Probability Embedding Vector), and then use it together with the initial placement state information of the objects already placed in the current room (such as the color, quantity, type, position, etc. of the placed objects) as the input of the placement evaluation model. Then, the placement evaluation model can predict the Score net and Energy net of the shape corresponding to the target placement state image. In addition, the placement evaluation model can also be used to generate a placement state combination that matches the target state information of the target placement state image according to the initial state information of the target placement space, forming a corresponding placement state combination set.

[0093] Specifically, Figure 4 As shown, there are 3 different colors (green-red-blue) of objects (balls) placed in the target placement space, and the number of objects of each color is 7. The initial state information (color, type, quantity, position, etc.) of a total of 21 objects can be used as input data for the above-mentioned placement evaluation model. At the same time, the target placement state image of the shape of "ε" composed of lines of 3 different colors (green-orange-blue) is also used as input data for the above-mentioned placement evaluation model. The first evaluation model S of the placement evaluation model can perform placement according to the initial state information such as color, type, quantity, position, etc. of the 21 target placement objects corresponding to the target placement space, with reference to the different colors and different shapes of the "ε" shape of the target placement state image, thereby generating multiple placement state combinations to form a corresponding placement state combination set.

[0094] Furthermore, corresponding Gaussian noise is generated for each placement state combination, and these Gaussian noises can be denoised in the process of generating the above placement state combination through the first evaluation model S to generate a placement state combination that meets the target placement shape. For example, if a batch of 36 target placement combinations of "ε" target placement shapes are to be generated, 36 Gaussian noises are first randomly generated, and then these Gaussian noises are denoised using the first evaluation model S to generate 36 target placement combinations that meet the "ε" target placement shape, thereby ensuring the accuracy of the subsequent valuation process; further, the second evaluation model E can be used to perform valuation processing on each placement state combination generated in the denoising process, thereby achieving the valuation effect of the discrete states at different stages.

[0095] Therefore, the placement evaluation model can not only learn and generalize object placement across multiple shapes, but also directly perform diversified object placement according to the placement shape defined by the input target placement state image, and can perform diversified placement of various placement shapes that have not been seen in recognition training. Moreover, the second evaluation model is used to realize the valuation of the corresponding value range, so that the placement shape defined by the target placement state image can be infinitely close to the placement shape defined by the target placement state image during the placement execution process, and the accuracy of object placement can be judged for various shapes (even including untrained shapes), and the generated multiple placement state combinations can be screened for accuracy. In addition, according to the above-mentioned placement evaluation model, a certain number of placements of various shapes can also be learned, so as to ensure that accurate placement can be performed according to hand-drawn graphics, with a higher degree of intelligence.

[0096] like Figure 2-Figure 4 As shown, according to one embodiment of the present invention, in operation S203, generating a placement process data set according to the placement state data set includes: performing sorting according to the estimated value of each placement state data in the placement state data set to generate a placement process data set.

[0097] Each placement state data has a corresponding "placement state combination: valuation" relationship, such as the placement state combination of the aforementioned 6 chairs, ABCDEF: 0.2, AFBCDE: 0.6, ECFDBA: 0.7, etc. The size of the valuation is used to sort the placement state combinations to generate a placement process data set. The placement process data set can be a data set of placement state combinations formed according to the valuation from small to large or from large to small.

[0098] The generation process of the placement process data set can also screen out the placement combinations that do not meet the threshold requirements based on a certain valuation threshold, so that only the placement combinations that meet the threshold requirements and the corresponding valuations are left as a subset of the placement process data set. Furthermore, the placement process data set can also select the placement combination with the largest valuation value as the only data of the placement process data set based on the valuation of each placement combination in the placement state data set, that is, in the execution of the placement process, directly place it from the initial placement state to the target placement state defined by the placement combination with the largest valuation value.

[0099] Therefore, according to the valuation of the placement evaluation model, the placement scoring of the target object can be realized to ensure the dynamic placement process from the chaotic initial state to the final state of the target shape.

[0100] In summary, the above-mentioned image processing method applied to object placement according to the embodiment of the present invention can realize an object placement process with greater adaptability, flexibility and intelligence without the need for additional data to train the reward function of the object placement process. Moreover, for the placement evaluation model of the embodiment of the present invention, after training, the process of guiding the placement of objects through gradients does not require path learning data, and does not require a special definition and description of the target placement state, and the overall intelligence level is higher. Moreover, through the placement evaluation model, objects can be placed according to the target placement shape defined by the input target placement state image, and the placement of objects with new shape input has a stronger generalization ability, unlike traditional reinforcement learning where a model can usually only complete a specific task.

[0101] Therefore, the image processing method for object placement provided by the embodiment of the present invention generates a placement state data set that matches the target state information through a placement evaluation model, and can realize valuation of each placement state data, so that the object placement effect can be evaluated according to the valuation, and the effective screening of the placement effect can be realized, thereby significantly improving the flexibility of placement in the object placement process, preventing the influence of complex environment, and greatly reducing the placement design and matching costs. At the same time, it also avoids the situation in which the objective function (such as reward function) in the prior art is difficult to establish, and can generalize the object placement of various application scenarios. At the same time, it can meet the placement requirements of irregular and diverse placement shapes (the placement shapes hand-drawn by the user can also achieve accurate and efficient placement effects), and can ensure the placement of diversified objects with higher precision, which substantially improves the placement efficiency and intelligence level of the placement process.

[0102] Based on the above-mentioned image processing method for object placement, the present invention also provides an image processing device for object placement. Figure 5 The device is described in detail.

[0103] Figure 5 The structure block diagram of the image processing device applied to object placement according to an embodiment of the present invention is schematically shown.

[0104] like Figure 5 As shown, the image processing device 500 applied to object placement in this embodiment includes an information acquisition module 510 , an information evaluation module 520 and a data generation module 530 .

[0105] The information acquisition module 510 is used to acquire the target state information of the target placement state image. In one embodiment, the information acquisition module 510 can be used to perform the operation S201 described above, which will not be described in detail here.

[0106] The information evaluation module 520 is used to generate a placement state data set that matches the target state information through a placement evaluation model; wherein the placement evaluation model is a neural network model that estimates the target state information based on the initial state information, and the initial state information is the initial placement state information of the target placement space. In one embodiment, the information evaluation module 520 can be used to perform the operation S202 described above, which will not be repeated here.

[0107] The data generation module 530 is used to generate a placement process data set according to the placement state data set, and the placement process data set is applied to the object placement. In one embodiment, the data generation module 530 can be used to perform the operation S203 described above, which will not be repeated here.

[0108] According to an embodiment of the present invention, any multiple modules among the information acquisition module 510, the information evaluation module 520 and the data generation module 530 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the information acquisition module 510, the information evaluation module 520 and the data generation module 530 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the information acquisition module 510, the information evaluation module 520 and the data generation module 530 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0109] Figure 6 A block diagram of an electronic device suitable for implementing an image processing method applied to object placement according to an embodiment of the present invention is schematically shown.

[0110] The electronic device provided by an embodiment of the present invention includes one or more processors and a memory, and the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned image processing method applied to object placement.

[0111] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0112] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0113] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0114] The present invention also provides a computer-readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the above-mentioned image processing method applied to object placement.

[0115] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0116] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0117] An embodiment of the present invention also includes a computer program product, which includes a computer program, and when the computer program is executed by a processor, the image processing method applied to object placement is implemented.

[0118] The computer program includes program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0119] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0120] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0121] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0122] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0125] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. An image processing method, applied to object placement, characterized in that: include: Obtain target state information of the target placement state image; Generate a placement state data set matching the target state information through a placement evaluation model, including: generating a placement state combination set matching the target state information according to initial state information through a first evaluation model of the placement evaluation model; performing valuation on each placement state combination of the placement state combination set through a second evaluation model of the placement evaluation model to generate the placement state data set; wherein the placement evaluation model is a neural network model that estimates the target state information according to initial state information, and the initial state information is initial placement state information of the target placement space; and Generating a placement process data set according to the placement state data set, which includes: performing sorting according to the estimated value of each placement state data in the placement state data set to generate the placement process data set; and applying the placement process data set to object placement.

2. The method according to claim 1, characterized in that The target state information of the target placement state image is obtained, including: The feature vector data of the target placement state image is extracted as the target state information.

3. The method according to claim 1, characterized in that Before obtaining the target state information of the target placement state image, the method includes: Obtain a feature vector dataset of a training image set; The placement assessment model is generated according to the feature vector data set.

4. The method according to claim 3, characterized in that The step of obtaining a feature vector data set of a training image set includes: Extracting vectorized data of each image in the training image set; Classification is performed on each image in the training image set according to the vectorized data to generate a corresponding feature vector data set.

5. The method according to claim 3, characterized in that: In generating the placement evaluation model according to the feature vector data set, comprising: Generate a first evaluation model based on the first neural network according to the feature vector data set; generating a second evaluation model based on a second neural network according to the feature vector data set; Wherein, the placement evaluation model includes the first evaluation model and the second evaluation model.

6. An image processing device for placing objects, characterized in that: include: An information acquisition module, used to acquire target state information of the target placement state image; An information evaluation module is used to generate a placement state data set matching the target state information through a placement evaluation model, including: generating a placement state combination set matching the target state information according to initial state information through a first evaluation model of the placement evaluation model; performing valuation on each placement state combination of the placement state combination set through a second evaluation model of the placement evaluation model to generate the placement state data set; wherein the placement evaluation model is a neural network model that estimates the target state information according to initial state information, and the initial state information is initial placement state information of the target placement space; and A data generation module is used to generate a placement process data set according to the placement state data set, which includes: performing sorting according to the estimated value of each placement state data in the placement state data set to generate the placement process data set; and the placement process data set is applied to object placement.

7. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images