Image data generation method, device, and electronic device

Abstract

Embodiments of the present application provide a method and apparatus for generating image data, and an electronic device. The method comprises: obtaining a first image containing a foreground image and a background image, the background image containing a plurality of artificial markers, and a projection of an object in the foreground image in the background image being located in an area surrounded by the plurality of artificial markers; calculating a spatial pose of the object relative to the camera according to the first image; mapping each pixel of a target image to the background image according to a spatial relationship between the artificial markers and the background image to generate a second image; generating a mask image using a three-dimensional model of the object and the spatial pose; and synthesizing a third image according to the second image and the mask image. Thus, the background of the object to be photographed can be marked using a plurality of artificial markers, and the background image of the object in the photographed image can be replaced with the target image, so that a large amount of image data close to an actual use scenario can be synthesized at a lower cost.

Description

Technical Field

[0001] This application relates to the field of image data generation, and more particularly to a method, apparatus, and electronic device for generating image data. Background Technology

[0002] In industrial production, robotic arms are widely used on production lines for picking up and placing objects. When using robotic arms to pick up objects, the spatial orientation information of the objects on the production line is particularly important for quick and accurate grasping. Therefore, it is necessary to label the spatial orientation of the objects.

[0003] With the development of machine learning technology, deep learning techniques are typically used to train AI (Artificial Intelligence) models, which are then applied to robotic arms, enabling them to accurately grasp objects. To ensure the AI ​​model has generalizability, a large amount of image data from different scenarios is required during training.

[0004] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention

[0005] The inventors discovered that in existing technologies, image data used to train models is divided into "real data" and "fake data." "Real data" is usually actual image data, but because it requires a large amount of image data, acquiring "real data" is time-consuming, labor-intensive, and has high operating costs. Furthermore, the accuracy of a model trained in a specific scene will decrease after changing the scene, meaning that the model's generalization ability is not high. "Fake data" is usually image data generated through image synthesis technology, but existing image synthesis methods only involve synthesizing two-dimensional images and do not involve techniques for changing the background of objects in images that include three-dimensional objects.

[0006] To address at least one of the above-mentioned problems or other similar problems, embodiments of this application provide a method and apparatus for generating image data.

[0007] According to a first aspect of the present application, an image data generation method is provided, wherein the generation method includes: acquiring a first image captured by a camera device, comprising a foreground image and a background image, wherein the background image contains a plurality of artificial markers, and the projection of an object in the foreground image onto the background image is located in the area surrounded by the plurality of artificial markers; calculating the spatial pose of the object in the foreground image relative to the camera device based on the first image; mapping each pixel of the target image onto the background image to generate a second image based on the spatial relationship between the plurality of artificial markers and the background image; generating a mask image corresponding to the first image using a three-dimensional model of the object in the foreground image and the spatial pose of the object relative to the camera device; and synthesizing a third image based on the second image and the mask image.

[0008] In at least one embodiment, calculating the spatial pose of the object in the foreground image relative to the camera device based on the first image includes: calculating a first spatial relationship between the coordinate system of the camera device and the coordinate system of any one of the plurality of artificial markers based on the first image; and calculating the spatial pose of the object relative to the camera device based on the first spatial relationship and a second spatial relationship between the object in the foreground image and the plurality of artificial markers.

[0009] In at least one embodiment, the generation method further includes: calculating a second spatial relationship between the coordinate system of the object in the foreground image and any one of the plurality of artificial markers based on the first image, and calculating the spatial pose based on the first spatial relationship and the second spatial relationship.

[0010] In at least one embodiment, the background image includes at least four artificial markers with different identification codes, the at least four artificial markers being located at at least four different positions in the background image.

[0011] In at least one embodiment, mapping each pixel of the target image to the background image to generate the second image includes: calculating a mapping matrix from the background image to the target image based on the position information of the plurality of artificial markers in the coordinate system of the background image, and generating the second image according to formula (1).

[0012]

[0013] in, For the pixels in the background image, Let H be the pixel points of the target image, and let H be the mapping matrix.

[0014] According to a second aspect of the present application, an image data generation apparatus is provided, comprising: an image acquisition unit that acquires a first image captured by a camera device, including a foreground image and a background image, wherein the background image contains a plurality of artificial markers, and the projection of an object in the foreground image onto the background image is located in the area surrounded by the plurality of artificial markers; a first calculation unit that calculates the spatial pose of the object in the foreground image relative to the camera device based on the first image; a mapping unit that maps each pixel of a target image onto the background image based on the spatial relationship between the plurality of artificial markers and the background image, thereby generating a second image; a mask generation unit that generates a mask image corresponding to the first image using a three-dimensional model of the object in the foreground image and the spatial pose of the object relative to the camera device; and a synthesis unit that synthesizes a third image based on the second image and the mask image.

[0015] In at least one embodiment, the first calculation unit includes: a first spatial relationship calculation unit, which calculates a first spatial relationship between the coordinate system of the camera device and the coordinate system of any one of the plurality of artificial marks based on the first image; and a spatial pose calculation unit, which calculates the spatial pose of the object relative to the camera device based on the first spatial relationship and a second spatial relationship between the object in the foreground image and the plurality of artificial marks.

[0016] In at least one embodiment, the first calculation unit further includes: a second spatial relationship calculation unit, which calculates a second spatial relationship between the object in the foreground image and any one of the plurality of artificial markers based on the first image, and the spatial pose calculation unit calculates the spatial pose based on the first spatial relationship calculated by the first spatial relationship calculation unit and the second spatial relationship calculated by the second spatial relationship calculation unit.

[0017] In at least one embodiment, the background image includes at least four artificial markers with different identification codes, the at least four artificial markers being located at at least four different positions in the background image.

[0018] In at least one embodiment, the mapping unit includes: a mapping matrix calculation unit, which calculates a mapping matrix from the background image to the target image based on the position information of the plurality of artificial markers in the coordinate system of the background image; and a second image generation unit, which generates a second image according to formula (1).

[0019]

[0020] in, For the pixels in the background image, Let H be the pixel points of the target image, and let H be the mapping matrix.

[0021] According to a third aspect of the present application, an electronic device is provided, wherein the electronic device includes a memory and a processor, the memory storing a computer program, and the processor is configured to execute the computer program to implement the image data generation method described in the first aspect of the present application.

[0022] One of the beneficial effects of this application's embodiments is that by using multiple artificial markers to mark the background of the object to be photographed, the background image of the object in the photographed image can be replaced by the target image, thereby synthesizing a large amount of image data that is close to the actual use scenario at a lower operating cost.

[0023] Specific embodiments of this application are disclosed in detail with reference to the following description and accompanying drawings, indicating how the principles of this application can be adopted. It should be understood that the embodiments of this application are not limited in scope. Within the scope of the appended claims, embodiments of this application include many changes, modifications, and equivalents.

[0024] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0025] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, or components. Attached Figure Description

[0026] The elements and features described in one drawing or embodiment of this application may be combined with elements and features shown in one or more other drawings or embodiments. Furthermore, in the drawings, similar reference numerals denote corresponding parts in several drawings and can be used to indicate corresponding parts used in more than one embodiment.

[0027] The accompanying drawings, which form part of the specification, are used to provide a further understanding of the embodiments of this application and illustrate the implementation methods of this application, together with the textual description, to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings:

[0028] Figure 1This is a flowchart illustrating a method for generating image data according to an embodiment of this application.

[0029] Figure 2 This is a schematic diagram of the first image of an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of the step of calculating the spatial pose of an object in the foreground image relative to the camera device in the generation method of this application embodiment.

[0031] Figure 4 It is in Figure 3 This is a schematic diagram illustrating the steps of calculating the spatial pose of an object in the foreground image relative to the camera device.

[0032] Figure 5 This is a schematic diagram of the mask image corresponding to the first image in an embodiment of this application.

[0033] Figure 6 This is a schematic diagram of the third image in an embodiment of this application.

[0034] Figure 7 This is a schematic diagram of an image data generation apparatus according to an embodiment of this application.

[0035] Figure 8 This is a schematic diagram of the first computing unit in an embodiment of this application.

[0036] Figure 9 This is a schematic diagram of a mapping unit in an embodiment of this application.

[0037] Figure 10 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0038] Referring to the accompanying drawings, the foregoing and other features of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of this application can be adopted. It should be understood that this application is not limited to the described embodiments; rather, it includes all modifications, variations, and equivalents falling within the scope of the appended claims. Various embodiments of this application are described below with reference to the accompanying drawings. These embodiments are merely exemplary and not intended to limit the scope of this application.

[0039] In the embodiments of this application, the terms "first," "second," "upper," "lower," etc., are used to distinguish different elements by their names, but do not indicate the spatial arrangement or temporal order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in connection with the application and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0040] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.

[0041] First aspect of the embodiments

[0042] An embodiment of the first aspect of this application provides a method for generating image data. Figure 1 This is a flowchart illustrating a generation method according to an embodiment of this application.

[0043] like Figure 1 As shown, the generation method 100 of this application embodiment may include the following steps:

[0044] Step 101: Acquire a first image, including a foreground image and a background image, captured by a camera device;

[0045] Step 103: Calculate the spatial pose of the object in the foreground image relative to the camera device based on the first image;

[0046] Step 105: Based on the spatial relationship between the multiple artificial markers and the background image, map each pixel of the target image to the background image to generate the second image;

[0047] Step 107: Generate a mask image corresponding to the first image using the 3D model of the object in the foreground image and the spatial pose of the object relative to the camera device; and

[0048] Step 109: Synthesize the third image based on the second image and the mask image.

[0049] It is worth noting that the above appendix Figure 1The embodiments of this application have been illustrated only, and the application is not limited thereto. For example, the execution order between the various steps can be appropriately adjusted, and other steps can be added or some operations can be reduced. Those skilled in the art can make appropriate modifications based on the above description, and are not limited to the above-described embodiments. Figure 1 The records.

[0050] In at least one embodiment, when capturing a first image of an object using a camera device, the background of the object can be marked with multiple artificial markers, and the projection of the object in the captured image is located in the area surrounded by the multiple artificial markers. Thus, the captured first image includes a foreground image and a background image, wherein the background image contains multiple artificial markers, and the projection of the object in the foreground image in the background image is located in the area surrounded by the multiple artificial markers.

[0051] Figure 2 This is a schematic diagram of the first image of an embodiment of this application. For example... Figure 2 As shown, the first image 200 includes a background image 201 and a foreground image 202. The background image 201 contains multiple artificial markers 203. The object in the foreground image 202 is located in the area surrounded by the multiple artificial markers 203. For ease of description, the object in the foreground image 202 is sometimes referred to by the reference numeral "202".

[0052] in addition, Figure 2 The artificial tag 203 shown is the ArUco (Augmented Reality University of Cordoba) tag. Alternatively, other artificial tags with similar functions can be used in this embodiment, such as the Apriltag. This embodiment does not limit the specific artificial tag used.

[0053] ArUco markers consist of a one-unit-width black border and a two-dimensional pattern within the border. The black border facilitates rapid marker detection in the image, while the internal two-dimensional pattern represents a binary matrix corresponding to the marker's identifier (i.e., its ID). Since each two-dimensional pattern is unique, each ArUco marker's ID is also unique. Furthermore, the binary matrix of the ArUco marker is known; therefore, information related to the position and orientation of the camera device can be obtained using the ArUco markers in the image. Additionally, the size and dimensions of the ArUco markers can be predefined according to actual needs; specific details can be found in relevant technologies.

[0054] Figure 3This is a schematic diagram of the step of calculating the spatial pose of an object in the foreground image relative to the camera device in the generation method of this application embodiment. Figure 4 It is in Figure 3 This diagram illustrates the steps involved in calculating the spatial pose of an object in the foreground image relative to the camera device. The following is an explanation of these steps. Figure 3 and Figure 4 The method for calculating the spatial pose of an object in a foreground image relative to a camera device in this application embodiment will be described using an example.

[0055] like Figure 3 As shown, step 103 may include the following steps:

[0056] Step 1031: Calculate the first spatial relationship between the coordinate system of the camera device and the coordinate system of any one of the multiple artificial marks based on the first image;

[0057] Step 1032: Calculate the second spatial relationship between the object in the foreground image and any one of the multiple artificial markers based on the first image;

[0058] Step 1033: Calculate the spatial pose of the object in the foreground image relative to the camera device based on the first spatial relationship and the second spatial relationship.

[0059] like Figure 4 As shown, the coordinate system of the camera device 400 is denoted as A. Multiple ArUco markers can be set in the background. The object 402 is located in the area enclosed by multiple ArUco markers. The coordinate system of any one of the ArUco markers 401 is denoted as B, and the coordinate system of the object 402 is denoted as C. The spatial pose of the object 402 in the foreground image relative to the camera device 400 can be calculated according to formula (11).

[0060]

[0061] in, This is the first spatial relation. This is the second spatial relationship. and These are the rotation and translation matrices of object 402 relative to camera device 400, respectively. In other words, and The spatial orientation of object 402 relative to camera device 400, for example, spatial information with six degrees of freedom.

[0062] In step 1031, since the positions of the camera device 400 and the ArUco marker 401 are preset when the first image is captured, that is, the relative positions of the camera device 400 and the ArUco marker 401 are known, the rotation matrix R and translation matrix T between the coordinate system A of the camera device 400 and the coordinate system B of the ArUco marker 401 are known. Therefore, it is possible to... The first spatial relation is calculated.

[0063] In step 1032, since the relative positions of object 402 and ArUco marker 401 in the foreground image are preset when the first image is captured, and if ArUco marker 401 and object 402 are relatively stationary, then there is no relative rotation between the coordinate system B of ArUco marker 401 and the coordinate system C of object 402. Therefore, the rotation matrix R between the coordinate system B of ArUco marker 401 and the coordinate system C of object 402 can be considered as an identity matrix, for example, R is a 3×3 identity matrix. Therefore, the translation matrix T between the coordinate system B of ArUco marker 401 and the coordinate system C of the three-dimensional object 402 can be obtained through actual measurement, thereby enabling [the calculation of the translation matrix T]. The second spatial relationship is calculated. Furthermore, since the relative positions of object 402 and ArUco marker 401 in the foreground image are preset, the second spatial relationship can also be predetermined.

[0064] In step 1033, the spatial orientation of object 402 relative to camera device 400 is calculated by formula (11) based on the first spatial relationship and the second spatial relationship.

[0065] In at least one embodiment, such as Figure 2 As shown, the background image 201 includes four artificial markers 203 with different IDs. These four artificial markers 203 are located at four different positions within the background image 201. For example, the four artificial markers can be placed at the four corners of the background image, thereby accurately replacing individual pixels in the background image. However, this embodiment is not limited to this; the artificial markers can be placed at any different position within the background image. Furthermore, this embodiment does not limit the number of artificial markers; it can be five or more. The more artificial markers there are, the more position and pose information can be obtained during calculation, resulting in more accurate calculation results. However, the computational load also increases. Therefore, an appropriate number of artificial markers can be selected based on actual needs.

[0066] In at least one embodiment, since the positions of the multiple artificial markers in the background image are known, the position information of the multiple artificial markers in the coordinate system of the background image can be known in advance. In step 105, a mapping matrix from the background image to the target image can be calculated based on the known position information, and then the second image is generated according to formula (1).

[0067]

[0068] in, For pixels in the background image, Let H be the pixels of the target image, and H be the mapping matrix.

[0069] For example, in Figure 2 In the scenario shown, a corner point is selected from each of the four artificial markers 203. For example, from one corner point of an artificial marker 203, other corner points of artificial markers 203 are selected in a clockwise direction. The corner point of the artificial marker is taken as the target point (dst), and the four corner points of the background image are taken as the source point (src). The homography function h between the source point and the target point is calculated, thereby solving the mapping matrix H. Then, the target image is mapped to the position of the background image in the first image through formula (1), thereby generating the second image.

[0070] In addition, other methods can be used to calculate the mapping matrix and generate the second image. For specific methods, please refer to relevant technologies. This application does not limit the specific methods used.

[0071] In at least one embodiment, the three-dimensional model of the object can be obtained in advance. The method for obtaining the three-dimensional model of the object can refer to related technologies, and this application embodiment does not limit it. In step 107, a mask image corresponding to the first image can be generated based on the spatial pose of the object relative to the camera device calculated in step 103 and the pre-obtained three-dimensional model of the object. The method for generating the mask image can refer to related technologies, and this application embodiment does not limit it.

[0072] Figure 5 This is a schematic diagram of the mask image corresponding to the first image in an embodiment of this application. For example, in step 107, the mask image is generated... Figure 5 The mask image shown.

[0073] In step 109, a third image is synthesized based on the second image generated in step 105 and the mask image generated in step 107. For example, a matrix operation is performed on the second image and the mask image to replace the background image in the first image with the target image while retaining the object in the foreground image. This makes the object appear as if it were photographed in a new scene, i.e., "fake data" is generated.

[0074] In addition, other methods can be used to synthesize the third image. For specific synthesis methods, please refer to relevant technologies. This application does not limit the specific methods used.

[0075] Figure 6 This is a schematic diagram of the third image in an embodiment of this application. For example, in step 109, the image is generated... Figure 6 The third image shown is the synthesized "fake data." Therefore, by replacing the background image in an image with an image that closely resembles the actual usage scenario, image data that closely resembles the actual usage scenario can be generated.

[0076] In addition, a large number of target images can be obtained by shooting videos of actual usage scenarios. Thus, "fake data" can be generated in batches by processing the video data.

[0077] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0078] As can be seen from the above embodiments, by using multiple artificial markers 203 to mark the background of the object 202 to be photographed, the background image of the object in the photographed image can be replaced by the target image, thereby generating a large amount of image data that is close to the actual use scenario at a low operating cost.

[0079] Second aspect of the embodiments

[0080] The second aspect of this application provides an image data generation apparatus. Since the principle of this generation apparatus in solving the problem is similar to the generation method of the first aspect embodiment, its specific implementation can refer to the embodiment of the method of the first aspect embodiment. Where the content is the same, it will not be described again.

[0081] Figure 7 This is a schematic diagram of the generation apparatus according to an embodiment of this application, as shown below. Figure 7 As shown, the generation device 700 may include an image acquisition unit 701, a first calculation unit 702, a mapping unit 703, a mask generation unit 704, and a synthesis unit 705.

[0082] The image acquisition unit 701 acquires a first image captured by a camera device, which includes the foreground image and the background image. The background image contains multiple artificial markers, and the projection of an object in the foreground image onto the background image is located in the area surrounded by the multiple artificial markers. The first calculation unit 702 calculates the spatial pose of the object in the foreground image relative to the camera device based on the first image. The mapping unit 703 maps each pixel of the target image onto the background image based on the spatial relationship between the multiple artificial markers and the background image, generating a second image. The mask generation unit 704 generates a mask image corresponding to the first image using the three-dimensional model of the object in the foreground image and the spatial pose of the object relative to the camera device. The synthesis unit 705 synthesizes a third image based on the second image and the mask image.

[0083] In at least one embodiment, when capturing a first image of an object using a camera device, the background of the object can be marked using multiple artificial markers, and the projection of the object in the captured image is located in the area surrounded by the multiple artificial markers. Thus, the first image acquired by the image acquisition unit 701 includes a foreground image and a background image, the background image containing multiple artificial markers, and the projection of the object in the foreground image in the background image located in the area surrounded by the multiple artificial markers.

[0084] Figure 8 This is a schematic diagram of the first computing unit 702 in an embodiment of this application.

[0085] In at least one embodiment, such as Figure 8 As shown, the first calculation unit 702 may include a first spatial relationship calculation unit 721 and a spatial pose calculation unit 722, wherein the first spatial relationship calculation unit 721 calculates a first spatial relationship between the coordinate system of the camera device and the coordinate system of any one of the plurality of artificial marks based on the first image; the spatial pose calculation unit 722 calculates the spatial pose of the object relative to the camera device based on the first spatial relationship and a second spatial relationship between the object in the foreground image and the plurality of artificial marks.

[0086] In at least one embodiment, such as Figure 8 As shown, the first calculation unit 702 may further include a second spatial relationship calculation unit 723. The second spatial relationship calculation unit 723 calculates the second spatial relationship between the object in the foreground image and any one of the multiple artificial markers in the first image based on the first image. The spatial pose calculation unit 722 calculates the spatial pose based on the first spatial relationship calculated by the first spatial relationship calculation unit 721 and the second spatial relationship calculated by the second spatial relationship calculation unit 723.

[0087] The methods for calculating the first spatial relationship, the second spatial relationship, and the spatial pose of a three-dimensional object relative to the camera device using the first spatial relationship and the second spatial relationship have been described in detail in the embodiments of the first aspect, the contents of which are incorporated herein.

[0088] Figure 9 This is a schematic diagram of the mapping unit 703 in an embodiment of this application.

[0089] In at least one embodiment, such as Figure 9 As shown, the mapping unit 703 may include a mapping matrix calculation unit 731 and a second image generation unit 732, wherein the mapping matrix calculation unit 731 calculates a mapping matrix from the background image to the target image based on the position information of the plurality of artificial markers in the coordinate system of the background image; the second image generation unit 732 generates a second image according to formula (1).

[0090]

[0091] in, For the pixels in the background image, Let H be the pixel points of the target image, and let H be the mapping matrix.

[0092] In at least one embodiment, since the positions of the multiple artificial markers in the background image are known, the position information of the multiple artificial markers in the coordinate system of the background image can be known in advance. The mapping matrix calculation unit 731 can calculate the mapping matrix H from the background image to the target image based on the known position information. The second image generation unit 732 maps the target image to the position of the background image in the first image according to formula (1), thereby generating the second image.

[0093] In addition, other methods can be used to calculate the mapping matrix and generate the second image. For specific methods, please refer to relevant technologies. This application does not limit the specific methods used.

[0094] In at least one embodiment, the three-dimensional model of the object can be obtained in advance. The method for obtaining the three-dimensional model of the object can refer to related technologies, and this application embodiment does not limit this. The mask generation unit 704 generates a mask image corresponding to the first image based on the pre-obtained three-dimensional model of the object and the spatial pose of the object relative to the camera device in the foreground image calculated by the first calculation unit 702. The method for generating the mask image can refer to related technologies, and this application embodiment does not limit this.

[0095] In at least one embodiment, the synthesis unit 705 synthesizes a third image based on the second image generated by the mapping unit 703 and the mask image generated by the mask generation unit 704. For example, by performing an addition operation between the second image and the mask image, the background image in the first image is replaced with the target image while retaining the object in the foreground image. This makes the object appear as if it were photographed in a new scene, i.e., "fake data" is generated. Therefore, the background image in an image can be replaced with an image that closely resembles the actual usage scenario, thereby generating image data that closely resembles the actual usage scenario.

[0096] In addition, other methods can be used to synthesize the third image. For specific synthesis methods, please refer to relevant technologies. This application does not limit the specific methods used.

[0097] In addition, a large number of target images can be obtained by shooting videos of actual usage scenarios. Thus, "fake data" can be generated in batches by processing the video data.

[0098] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0099] It is worth noting that the above description only covers the components or modules relevant to this application, but this application is not limited thereto. The generation apparatus 700 of this application embodiment may also include other components or modules, and for details regarding these components or modules, please refer to related technologies.

[0100] In addition, for the sake of simplicity, Figures 7 to 9 The various components or modules are only illustrated by way of example, and the connection relationships or signal routing between modules are omitted. However, those skilled in the art should understand that various related technologies such as bus connections can be used. The aforementioned components or modules can be implemented by hardware facilities such as processors, memory, transmitters, and receivers; the implementation of this application does not limit this.

[0101] As can be seen from the above embodiments, by using the generation device 700 to replace the background image containing artificial markers in the image data with the target image, a large amount of image data that is close to the actual use scenario can be generated at a low operating cost.

[0102] Third aspect of the embodiments

[0103] This application provides an electronic device, and the contents that are the same as those in the first and second aspects will not be repeated.

[0104] Figure 10This is a schematic diagram of an electronic device according to an embodiment of this application. Figure 10 As shown, the electronic device 800 may include a processor (e.g., a central processing unit, CPU) 801 and a memory 802; the memory 802 is coupled to the processor 801. The memory 802 may store various types of data; it also stores an information processing program 804, and executes the program 804 under the control of the central processing unit 801.

[0105] For example, processor 801 may be configured to execute a program to implement the generation method as described in the first aspect embodiment. For example, processor 801 may be configured to perform the following control: acquire a first image captured by a camera device, comprising a foreground image and a background image, wherein the background image contains a plurality of artificial markers, and the projection of an object in the foreground image onto the background image is located in the region enclosed by the plurality of artificial markers; calculate the spatial pose of the object in the foreground image relative to the camera device based on the first image; map each pixel of the target image onto the background image based on the spatial relationship between the plurality of artificial markers and the background image, generating a second image; generate a mask image corresponding to the first image using a three-dimensional model of the object in the foreground image and the spatial pose of the object relative to the camera device; and synthesize a third image based on the second image and the mask image.

[0106] In at least one embodiment, the processor 801 may be configured to perform the following control: calculate a first spatial relationship between the coordinate system of the camera device and the coordinate system of any one of the plurality of artificial markers based on the first image; and calculate the spatial pose of the object relative to the camera device based on the first spatial relationship and a second spatial relationship between the object in the foreground image and the plurality of artificial markers.

[0107] In at least one embodiment, the processor 801 may be configured to perform the following control: calculate a second spatial relationship between the object in the foreground image and any one of the plurality of artificial markers in the coordinate system based on the first image, and calculate the spatial pose based on the first spatial relationship and the second spatial relationship.

[0108] In at least one embodiment, the background image includes at least four artificial markers with different identification codes, the at least four artificial markers being located at at least four different positions in the background image.

[0109] In at least one embodiment, the processor 801 may be configured to perform the following control: calculate a mapping matrix from the background image to the target image based on the position information of the plurality of artificial markers in the coordinate system of the background image, and generate a second image according to formula (1).

[0110]

[0111] in, For the pixels in the background image, Let H be the pixel points of the target image, and let H be the mapping matrix.

[0112] In addition, such as Figure 10 As shown, the electronic device 800 may also include: an I / O module 803, etc.; the functions of the above components are similar to those in the prior art, and will not be described in detail here. It is worth noting that the electronic device 800 is not necessarily required to include... Figure 8 All components shown; in addition, the electronic device 800 may also include Figure 10 For components not shown, please refer to existing technologies.

[0113] This application also provides a computer-readable program, wherein when the program is executed in an electronic device, the program causes the computer to perform the generation method in the first aspect of the embodiment in the electronic device.

[0114] This application also provides a storage medium storing a computer-readable program, wherein the computer-readable program causes a computer in an electronic device to perform the generation method of the first aspect embodiment.

[0115] The apparatus and methods described above in this application can be implemented in hardware or in combination with software. This application relates to a computer-readable program that, when executed by a logic component, enables the logic component to implement the apparatus or components described above, or to implement the various methods or steps described above. Logic components include, for example, field-programmable logic devices (FPGAs), microprocessors, and processors used in computers. This application also relates to storage media for storing the above programs, such as hard disks, magnetic disks, optical disks, DVDs, and flash memory.

[0116] The methods / apparatus described in conjunction with the embodiments of this application can be directly embodied in hardware, software modules executed by a processor, or a combination of both. For example, one or more and / or combinations of one or more functional block diagrams shown in the figures can correspond to various software modules in a computer program flow, or to various hardware modules. These software modules can correspond to the various steps shown in the figures, respectively. These hardware modules can be implemented, for example, using a field-programmable gate array (FPGA) to embed these software modules.

[0117] The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor, enabling the processor to read information from and write information to the storage medium; or the storage medium can be an integral part of the processor. The processor and storage medium can reside in an ASIC. The software module can be stored in the memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the device (such as a mobile terminal) uses a high-capacity MEGA-SIM card or a high-capacity flash memory device, the software module can be stored in the MEGA-SIM card or the high-capacity flash memory device.

[0118] One or more and / or one or more combinations of functional blocks described in the accompanying drawings can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described herein. One or more and / or one or more combinations of functional blocks described in the accompanying drawings can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0119] The present application has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on its spirit and principles, and these modifications and variations are also within the scope of the present application.

Claims

1. A method of generating image data, characterized by, The generation method comprises: acquiring a first image containing a foreground image and a background image captured by a camera, wherein the background image contains a plurality of artificial markers, and a projection of an object in the foreground image in the background image is located in an area surrounded by the plurality of artificial markers; calculating a spatial pose of the object in the foreground image relative to the camera according to the first image; mapping each pixel of a target image into the background image according to a spatial relationship between the plurality of artificial markers and the background image to generate a second image; generating a mask image corresponding to the first image by using a three-dimensional model of the object in the foreground image and the spatial pose of the object relative to the camera; and synthesizing a third image according to the second image and the mask image, wherein the background image includes at least four artificial markers with different identification codes, and four artificial markers among the at least four artificial markers with different identification codes are respectively located at four corners in the background image, wherein the mapping each pixel of the target image into the background image to generate the second image comprises: calculating a mapping matrix from the background image to the target image according to position information of the plurality of artificial markers in a coordinate system of the background image, generating a second image according to formula (1), （1）， wherein, is a pixel point in the background image, is a pixel point of the target image, H is the mapping matrix, wherein one corner point of each of the four artificial markers is taken as a target point, four corner points of the background image are taken as source points, a homography function between the source points and the target points is calculated, and the mapping matrix is calculated according to the homography function.

2. The generation method of claim 1, wherein, The calculating the spatial pose of the object in the foreground image relative to the camera according to the first image comprises: calculating a first spatial relationship of a coordinate system of the camera relative to a coordinate system of any one of the plurality of artificial markers according to the first image; calculating the spatial pose of the object relative to the camera according to the first spatial relationship and a second spatial relationship of the object in the foreground image relative to the plurality of artificial markers.

3. The generation method of claim 2, wherein, The generation method further comprises: calculating a second spatial relationship of the object in the foreground image relative to a coordinate system of any one of the plurality of artificial markers according to the first image, calculating the spatial pose according to the first spatial relationship and the second spatial relationship.

4. An image data generating apparatus characterized by comprising: The generation device comprises: an image acquisition unit that acquires a first image containing a foreground image and a background image captured by a camera, wherein the background image contains a plurality of artificial markers, and a projection of an object in the foreground image in the background image is located in an area surrounded by the plurality of artificial markers; a first calculation unit that calculates a spatial pose of the object in the foreground image relative to the camera according to the first image; a mapping unit that maps each pixel of a target image into the background image according to a spatial relationship between the plurality of artificial markers and the background image to generate a second image; a mask generation unit that generates a mask image corresponding to the first image using a three-dimensional model of the object in the foreground image and a spatial pose of the object with respect to the camera device; and a synthesis unit that synthesizes a third image from the second image and the mask image, wherein the background image includes at least four artificial markers having different identification codes, and four of the at least four artificial markers having different identification codes are respectively located at four corners in the background image, wherein the mapping of each pixel of the target image to the background image to generate the second image includes: calculating a mapping matrix from the background image to the target image based on position information of the plurality of artificial markers in a coordinate system of the background image, generating a second image according to formula (1), （1）， wherein, is a pixel point in the background image, is a pixel point of the target image, H is the mapping matrix, wherein one corner point of each of the four artificial markers is taken as a target point, and four corner points of the background image are taken as source points, a homography function between the source points and the target points is calculated, and the mapping matrix is calculated based on the homography function.

5. The generating device of claim 4, wherein, The first calculation unit includes: a first spatial relationship calculation unit that calculates a first spatial relationship of a coordinate system of the camera device with respect to a coordinate system of any one of the plurality of artificial markers based on the first image; and a spatial pose calculation unit that calculates a spatial pose of the object with respect to the camera device based on the first spatial relationship and a second spatial relationship of the object in the foreground image with respect to the plurality of artificial markers.

6. The generating device of claim 5, wherein, The first calculation unit further includes: a second spatial relationship calculation unit that calculates a second spatial relationship of the object in the foreground image with respect to a coordinate system of any one of the plurality of artificial markers based on the first image, The spatial pose calculation unit calculates the spatial pose based on the first spatial relationship calculated by the first spatial relationship calculation unit and the second spatial relationship calculated by the second spatial relationship calculation unit.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the image data generation method of any one of claims 1 to 3.

Images