An image generation method, apparatus and electronic device

Abstract

This invention provides an image generation method, apparatus, and electronic device, applied in the field of 3D simulation technology. The method includes: acquiring a specified number of package models; simulating the motion of the specified number of package models on a preset package-carrying platform model in a 3D simulation space; rendering each package model and the package-carrying platform model from the perspective of a specified camera in the 3D simulation space to obtain a rendered image; and fusing the model regions in the rendered image with a pre-acquired logistics scene background image to obtain a package image. This solution can improve the efficiency of acquiring package images in logistics scenes.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional simulation technology, and in particular to an image generation method, apparatus, and electronic device. Background Technology

[0002] In logistics scenarios, in order to efficiently process packages on conveyor belts or packaging platforms, such as counting packages or providing package grabbing information for robotic arms, it is necessary to obtain package information from the conveyor belts or packaging platforms. This package information includes: the bounding box of the package in the package image and the pixel-by-pixel information within the bounding box, as well as information such as the package's depth and normal plane.

[0003] In related technologies, pre-trained deep learning algorithms are mainly used to process the acquired package images to obtain package information. Training deep learning algorithms requires a large number of sample images, while related technologies primarily collect sample images through on-site acquisition, which is inefficient. Summary of the Invention

[0004] The purpose of this invention is to provide an image generation method, apparatus, and electronic device to improve the efficiency of acquiring package images in logistics scenarios. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of the present invention provide an image generation method, the method comprising:

[0006] Get a specified number of package models;

[0007] In a three-dimensional simulation space, the motion process of the specified number of package models on a preset package carrying platform model is simulated.

[0008] After the simulated motion ends, the rendering images of each package model and the package carrying platform model are obtained from the perspective of a specified camera in the three-dimensional simulation space.

[0009] The model region in the rendered image is fused with a pre-collected logistics scene background image to obtain a package image, wherein the model region is the area where the package model and the package carrying platform model are located.

[0010] Optionally, simulating the motion of the specified number of package models on a preset package carrying platform model includes:

[0011] Determine the initial pose of the specified number of package models within the region represented by the package carrying platform model;

[0012] For each package model, on the package carrying platform model, the initial pose of the package model is used as the starting pose for motion simulation of the package model.

[0013] Optionally, the step of simulating motion of the package model on the package-carrying platform model, using the initial pose of the package model as the starting pose of motion, includes:

[0014] On the package-carrying platform model, the initial pose of the package model is used as the starting pose for motion, and a gravity simulation of the package model is performed for a specified duration.

[0015] Optionally, before fusing the model region in the rendered image with a pre-acquired logistics scene background image to obtain the package image, the method further includes:

[0016] Using the perspective of the designated camera in the three-dimensional simulation space, each package model after the motion is completed is colored and rendered to obtain a first colored image, wherein the color value of the area where the package model and the package carrying platform model are located in the first colored image is a first color value, and the color value of the background area in the first colored image is a second color value.

[0017] The smallest bounding region containing the first color value in the first stained image is determined as the model region.

[0018] Optionally, fusing the model region in the rendered image with a pre-collected logistics scene background image to obtain a package image includes:

[0019] Based on the model region in the first coloring map, a region mask for the model region is generated;

[0020] In the rendered image, the image region corresponding to the region mask is retained as the first image region, and the image region corresponding to the region mask is removed from the pre-collected logistics scene background image to obtain the second image region;

[0021] The first image region and the second image region are merged to obtain the wrapped image.

[0022] Optionally, after obtaining a specified number of package models, and before simulating the motion of the specified number of package models on a preset package-carrying platform model in a three-dimensional simulation space, the method further includes:

[0023] For each package model, select a package texture from the preset package textures, and then use the selected package texture to perform texture fitting on the package model.

[0024] Optionally, the specified number of wrapping models include: rigid wrapping models; each wrapping texture includes a rigid wrapping surface texture and a single-sided texture;

[0025] The step of selecting a wrapping texture for the wrapping model from a preset set of wrapping textures, and then using the selected wrapping texture to perform texture bonding on the wrapping model, includes:

[0026] Select the hard-wrapped surface texture and single-face texture for the hard-wrapped model from the various hard-wrapped surface textures and single-face textures.

[0027] Apply the selected hard-covered surface texture to the surface of the hard-covered model.

[0028] On the model surface that already has the selected hard-surface texture applied, paste the selected single-sided texture.

[0029] Optionally, the specified number of wrapping models include: flexible wrapping models; each wrapping texture includes a flexible wrapping surface texture;

[0030] The step of selecting a wrapping texture for the wrapping model from a preset set of wrapping textures, and then using the selected wrapping texture to perform texture bonding on the wrapping model, includes:

[0031] Select the flexible wrapping surface texture for this flexible wrapping model from the various flexible wrapping surface textures;

[0032] Apply the selected flexible wrapping surface texture to the surface of the flexible wrapping model.

[0033] Optionally, before rendering the package models and the package-carrying platform model after the motion is completed, using the viewpoint of a camera specified in the three-dimensional simulation space, to obtain a rendered image, the method further includes:

[0034] Set up the camera at a specified location in the three-dimensional simulation space according to the preset camera parameters;

[0035] Adjust the orientation of the set camera and use the camera with the adjusted orientation as the designated camera.

[0036] Optionally, adjusting the pose of the set camera includes:

[0037] Adjust the camera's orientation to the specified orientation; or, randomly adjust the camera's orientation.

[0038] Optionally, obtaining a specified number of package models includes:

[0039] Obtain package density information corresponding to the target logistics scenario;

[0040] Based on the package density information, the number of candidate package models is determined as the specified number;

[0041] Select the specified number of package models from the preset package models.

[0042] Optionally, the package density information indicates the maximum and / or minimum number of packages on the package carrying platform within the target logistics scenario;

[0043] The specified quantity is less than or equal to the maximum number of packages, and / or greater than or equal to the minimum number of packages.

[0044] Optionally, before fusing the model region in the rendered image with a pre-acquired logistics scene background image to obtain the package image, the method further includes:

[0045] The rendered image is determined to contain truth information about each package model.

[0046] Optionally, determining that the rendered image contains wrapping truth information for each wrapping model includes:

[0047] Using the perspective of the specified camera in the three-dimensional simulation space, each package model after the motion is completed is colored and rendered to obtain a second colored image. In the second colored image, the color value of the area where each package model is located is the same, and the color value of the area where different package models are located is different.

[0048] For each enclosed model in the stained image, the minimum enclosing region of the color value corresponding to the enclosed model is determined from the second stained image, and the coordinate information of each pixel within the minimum enclosing region is used as the ground truth information of the enclosed model.

[0049] Optionally, before rendering the package models and the package-carrying platform model from the perspective of a specified camera in the three-dimensional simulation space after the motion simulation ends, and before obtaining the rendered image, the method further includes:

[0050] Lighting simulation is performed in the three-dimensional simulation space; wherein, the lighting simulation includes conventional lighting simulation and / or interference lighting simulation.

[0051] In a second aspect, embodiments of the present invention provide an image generation apparatus, the apparatus comprising:

[0052] The model acquisition module is used to acquire a specified number of package models;

[0053] The motion simulation module is used to simulate the motion process of the specified number of package models on a preset package carrying platform model in a three-dimensional simulation space.

[0054] The image rendering module is used to render the package models and the package carrying platform model after the motion simulation ends, from the perspective of a specified camera in the three-dimensional simulation space, to obtain a rendered image.

[0055] The image fusion module is used to fuse the model region in the rendered image with a pre-collected logistics scene background image to obtain a package image, wherein the model region is the area where the package model and the package carrying platform model are located.

[0056] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0057] Memory, used to store computer programs;

[0058] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0059] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0060] Beneficial effects of the embodiments of the present invention:

[0061] This invention provides an image generation method, apparatus, and electronic device that can acquire a specified number of package models and simulate their motion on a preset package-carrying platform model in a three-dimensional simulation space. After the simulation ends, the method renders each package model and the package-carrying platform model from the perspective of a specified camera in the three-dimensional simulation space to obtain a rendered image. The method then fuses the model regions in the rendered image with a pre-acquired logistics scene background image to obtain a package image. Because the motion of the package models on the package-carrying platform model can be simulated to obtain a rendered image, and then fused with the logistics scene background image to obtain a package image, package images can be automatically generated, improving the efficiency of acquiring package images in logistics scenes.

[0062] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0064] Figure 1 A flowchart of an image generation method provided in an embodiment of the present invention;

[0065] Figure 2a A schematic diagram of a package model provided in an embodiment of the present invention;

[0066] Figure 2b This is a schematic diagram of another package model provided in an embodiment of the present invention;

[0067] Figure 3a This is a schematic diagram of a package carrying platform model provided in an embodiment of the present invention;

[0068] Figure 3b This is a schematic diagram of another package carrying platform model provided in an embodiment of the present invention;

[0069] Figure 4a This is a schematic diagram of the first type of wrapping texture provided in an embodiment of the present invention;

[0070] Figure 4b This is a schematic diagram of the second type of wrapping texture provided in an embodiment of the present invention;

[0071] Figure 4c This is a schematic diagram of the third type of wrapping texture provided in an embodiment of the present invention;

[0072] Figure 5 This is a schematic flowchart of an image generation method provided in an embodiment of the present invention;

[0073] Figure 6 This is a schematic diagram of the structure of an image generation device provided in an embodiment of the present invention;

[0074] Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0076] In logistics scenarios, in order to efficiently process packages on conveyor belts or packaging platforms, such as counting packages or providing package grabbing information for robotic arms, it is necessary to obtain package information from the conveyor belts or packaging platforms. This package information includes: the bounding box of the package in the package image and the pixel-by-pixel information within the bounding box, as well as information such as the package's depth and normal plane.

[0077] In related technologies, pre-trained deep learning algorithms are mainly used to process the acquired package images to obtain package information. Training deep learning algorithms requires a large number of sample images, while related technologies primarily collect sample images through on-site acquisition, which is inefficient.

[0078] Meanwhile, some related technologies also include methods that create package images by directly pasting existing package sample images onto a two-dimensional scene. However, because these methods cannot simulate the real physical modes of the package, such as its movement and stacking state, the quality of the acquired package images is low.

[0079] To address the technical problems existing in related technologies, embodiments of the present invention provide an image generation method, apparatus, and electronic device.

[0080] It should be noted that, in specific applications, the embodiments of the present invention can be applied to various electronic devices, such as personal computers, servers, mobile phones, and other devices with data processing capabilities. Furthermore, the image generation method provided by the embodiments of the present invention can be implemented through software, hardware, or a combination of both.

[0081] The image generation method provided in this embodiment of the invention may include:

[0082] Get a specified number of package models;

[0083] In a three-dimensional simulation space, the motion process of a specified number of package models on a preset package carrying platform model is simulated.

[0084] After the simulation ends, the rendering images are obtained by rendering each package model and the package carrying platform model from the perspective of a specified camera in the three-dimensional simulation space.

[0085] The model region in the rendered image is fused with the pre-collected logistics scene background image to obtain the package image, where the model region is the area where the package model and the package carrying platform model are located.

[0086] In the above-described solution of this invention, since the package model can be simulated on the package carrying platform model to obtain a rendered image, and then combined with the logistics scene background image to obtain a package image, the package image can be automatically generated, thereby improving the efficiency of obtaining package images in the logistics scene.

[0087] The image generation method provided by the embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0088] like Figure 1 As shown, this embodiment of the invention provides an image generation method, including steps S101-S104, wherein:

[0089] S101, Get a specified number of package models;

[0090] In this scheme, the wrapping model can be divided into a rigid wrapping model and a flexible wrapping model. The rigid wrapping model is composed of regular cubes, while the flexible wrapping model is a deformation based on the rigid wrapping model.

[0091] For example, such as Figure 2a As shown in the diagram, this embodiment of the invention provides a schematic diagram of a wrapping model, including a hard wrapping rendering image and a hard wrapping model, as follows. Figure 2b As shown in the figure, an embodiment of the present invention provides a schematic diagram of another wrapping model, including a flexible wrapping rendering and a flexible wrapping model.

[0092] Optionally, 100 models of different sizes can be created for both the rigid and flexible wrapping models, denoted as Model. hard With Model soft It is important to emphasize that both the rigid and flexible wrapping models include material information for the wrapping model. This material information can include reflection parameters, refraction parameters, fading parameters, etc., indicating the shape exhibited by the wrapping model when illuminated. Specifically, the rigid wrapping model can contain two pieces of material information: one is the surface texture material of the wrapping. face_hard One is a single-sided textured material. bill The flexible wrapper can contain material information, which can be the surface texture material of the wrapper. face_soft .

[0093] The specified quantity can be any number, or it can be a number determined based on actual needs and experience. Optionally, to improve realism and increase the credibility of the acquired package images, the specified quantity can be determined based on the number of packages in the actual logistics scenario. In this case, one implementation can obtain the package density information corresponding to the target logistics scenario, and based on the package density information, determine the number of candidate package models as the specified quantity, and select the specified number of package models from the preset package models. Optionally, the package density information indicates the maximum and / or minimum number of packages on the package carrying platform within the target logistics scenario. In this case, the specified quantity is less than or equal to the maximum number of packages, and / or greater than or equal to the minimum number of packages.

[0094] In short, for a single simulation of a logistics scenario, an appropriate number of packages can be selected. When choosing a package model, a combination of rigid and flexible package models can be used, and then, based on the package density of different logistics scenarios, a minimum package quantity P can be selected. min With the maximum number of packages P max Random value P between num One package. The random value P num That is, the specified quantity.

[0095] S102, In a three-dimensional simulation space, simulate the motion process of a specified number of package models on a preset package carrying platform model;

[0096] The aforementioned three-dimensional simulation space can be a three-dimensional space constructed using a 3D engine. Correspondingly, the package model and the package-carrying platform model are both pre-constructed 3D models using a 3D engine. The package-carrying platform model can be pre-constructed and is mainly achieved through methods such as cutting, trimming, and splicing geometric solid shapes. After modeling is completed, the model needs to be color-rendered. In this embodiment of the invention, a Model can be used... plane This represents a parcel-carrying platform model. This parcel-carrying platform can be a conveyor belt model and / or a parcel-feeding station model. The conveyor belt, driven by a motor, is a tool used in logistics scenarios to transport express parcels, while the parcel-feeding station is a platform that provides carrying capacity when batches of parcels temporarily converge in a logistics scenario. For example, such as... Figure 3a As shown in the diagram, this embodiment of the invention provides a schematic diagram of a parcel carrying platform model, which includes a first type of parcel carrying platform model, such as... Figure 3b As shown in the figure, this embodiment of the invention provides a schematic diagram of another parcel carrying platform model, which includes a second type of parcel carrying platform model.

[0097] In one implementation, the initial poses of a specified number of package models within the designated region of the package-carrying platform model can be determined. Then, for each package model, motion simulation is performed on the package-carrying platform model using its initial pose as the starting pose for motion. Specifically, a gravity simulation of a specified duration can be performed on the package-carrying platform model using its initial pose as the starting pose for motion. The specified duration can be determined based on actual needs and experience, for example, 10 seconds.

[0098] To better initialize the package model, the pose initialization method used is to select horizontal and vertical ranges on the plane on which the package is carried by the package-bearing platform model, denoted as ±Loc respectively. x With ±Loc y Then, select the height range of the package [0, Loc] above this horizontal range. z The pose of the selected package model is randomly initialized in the above three-dimensional space to minimize collisions between packages, and the packages are randomly rotated.

[0099] After initialization, the physics simulation function in the 3D engine can be used to add gravity simulation to all the package models, causing them to fall and collide under the influence of gravity. Optionally, the fixed time length of the physics interaction on the package models can be set to T. phy During this period, all package models are free to undergo pose changes in the physical simulation. In T phy After the time limit expires, it's necessary to further filter packages based on their valid poses and delete those that are not in valid poses. In this step, the center position of a package is defined as not being within ±Loc in Step 3 on the horizontal plane. x and ±Loc y Packages within the range, and those not in the vertical direction [0, Loc z Packages within the specified range are considered illegal. In this step, all illegal packages will be deleted, and all legitimate packages will be retained.

[0100] S103, after the simulation motion ends, render the package models and package carrying platform models from the perspective of a specified camera in the three-dimensional simulation space to obtain the rendered images.

[0101] The designated camera in the 3D model space can be a camera that has been pre-set in the 3D model space. Optionally, the camera can be set at a designated position in the 3D simulation space according to the preset camera parameters, and then the attitude of the set camera can be adjusted, and the camera with the adjusted attitude can be used as the designated camera.

[0102] The above-mentioned adjustment of the camera's attitude can be done in various ways. For example, the camera's attitude can be adjusted to a specified attitude, which can be determined based on actual needs and experience, such as the attitude facing the package carrying platform model. Alternatively, the camera's attitude can be randomly adjusted.

[0103] Taking a stereo camera as an example, optionally, the stereo camera can be set in the 3D engine according to preset camera setting rules. The camera position is set above the package support platform model, and the fixed three-dimensional coordinates are represented as (C x C y C z After setting the camera position, the camera's attitude needs to be adjusted. Here, adjusting the camera attitude means rotating the camera angle. In the camera pose settings, the camera angle rotation is also three-dimensional, with rotation ranges set in the x, y, and z directions. The final camera rotation angle is a random value within this rotation range. (Set as (R...) x ,R y ,R z In this step, you can also set the camera parameters, including the camera's focal length in millimeters, the distance between the two eyes, the camera's field of view, the camera's minimum sensing distance, the camera's maximum sensing distance, and the scale of the image the camera captures.

[0104] After the camera settings are complete, you can render the various package models and the package-carrying platform model in the 3D simulation space from the specified camera's perspective to obtain rendered images. The above image rendering includes at least one of the following: RGB (Red-Green-Blue) image rendering, depth map rendering, and normal plane map rendering. First, set the world background for rendering to Color. bg Based on the specified scale of a single image captured by the camera, the three types of images mentioned above are rendered using the 3D engine's rendering interface. These are respectively represented as Image. rgb Image depth and Image normal .

[0105] S104, the model region in the rendered image is fused with the pre-collected logistics scene background image to obtain the package image, wherein the model region is the area where the package model and the package carrying platform model are located.

[0106] The pre-collected logistics scene background image can be a background image captured for actual logistics scenarios to make the final package image more realistic. The logistics scene background image refers to an image taken by a real camera in a logistics scenario, which may include the logistics package carrying platform, the package, and the background.

[0107] Before performing image fusion, the model region can be determined first. Optionally, the package models after motion can be colored and rendered from the perspective of a specified camera in the three-dimensional simulation space to obtain a first colored image. In the first colored image, the color value of the area where the package model and the package carrying platform model are located is the first color value, and the color value of the background area in the first colored image is the second color value. Then, the smallest bounding region in the first colored image is determined as the model region.

[0108] At this point, a region mask can be generated based on the model region in the first coloring image. Then, in the rendered image, the image region corresponding to the region mask is retained as the first image region. The image region corresponding to the region mask is removed from the pre-collected logistics scene background image to obtain the second image region. The first image region and the second image region are then merged to obtain the package image.

[0109] Specifically, a first tinted image can be rendered, followed by background blending. Alternatively, the tinting interface in the 3D engine can be used to tint all existing package models and package-carrying platform models simultaneously, coloring all package models and package-carrying platform models with Color. seg And render the image segmentation map, which serves as the first stained image. seg In the first stained image, the color of the package model and the package-carrying platform model is Color. seg The background color for the rest is Color. bg In terms of background fusion, one logistics scene background image is randomly selected from the pre-collected logistics scene background images. real As the background image for image generation, the fusion process requires the use of the first stained image. seg First, convert the RGB image... rgb Image with logistics scene background real All were converted into first-color images. seg The same scale, then the first stained image Image seg The model region in the image is used as a region mask to filter the real image. rgb Image with logistics scene background real In RGB image rgb The color in the first stained image is preserved. seg The area in the background image of the logistics scene. real Color of the middle retention area mask bg The two regions are then superimposed to obtain a fused image.fusion And save the Image fusion .

[0110] In the above-described solution of this invention, since the package model can be simulated on the package carrying platform model to obtain a rendered image, and then combined with the logistics scene background image to obtain a package image, the package image can be automatically generated, thereby improving the efficiency of obtaining package images in the logistics scene.

[0111] In one embodiment, in order to improve the realism of the acquired package images, after acquiring a specified number of package models and before simulating the motion process of the specified number of package models on a preset package carrying platform model in a three-dimensional simulation space, for each package model, a package texture for that package model is selected from preset package textures, and the selected package texture is used to perform texture bonding on the package model.

[0112] In one implementation, the specified number of wrapping models include rigid wrapping models; each wrapping texture includes a rigid wrapping surface texture and a surface texture. In this case, a rigid wrapping surface texture and a surface texture specific to the rigid wrapping model can be selected from the rigid wrapping surface texture and surface texture. The selected rigid wrapping surface texture is then applied to the surface of the rigid wrapping model, and the selected surface texture is pasted onto the surface of the model that has already been applied with the selected rigid wrapping surface texture.

[0113] In another implementation, the specified number of wrapping models include: flexible wrapping models; each wrapping texture contains a flexible wrapping surface texture. In this case, a flexible wrapping surface texture for the flexible wrapping model can be selected from the flexible wrapping surface textures, and then the selected flexible wrapping surface texture can be attached to the model surface of the flexible wrapping model.

[0114] In simple terms, when rendering package images in a logistics scene, a surface needs to be attached to the package model. Each surface texture can be bound to the material of the package model. There are two types of surface textures for rigid package models: one is the real-world surface texture Imageface_hard; the other is a single-face texture Imagebill. Flexible packages only have one type of surface texture, Imageuv_soft, which can be an unfolded image of the real-world flexible package surface. For example... Figure 4a As shown, the first type of wrapping texture diagram provided in this embodiment of the invention includes a schematic diagram of a rigid wrapping sheet texture; as... Figure 4b As shown, this is a schematic diagram of the second type of encapsulation texture provided in an embodiment of the present invention, which includes a schematic diagram of a rigid encapsulation surface texture; as shown Figure 4cThe diagram shown is a schematic diagram of the third type of wrapping texture provided in the embodiment of the present invention, which includes a schematic diagram of a flexible wrapping surface texture.

[0115] In the above-described solution of the present invention, package images can be automatically generated, which improves the efficiency of acquiring package images in logistics scenarios and enhances the authenticity of the final acquired package images.

[0116] In one embodiment, in order to obtain richer information, before fusing the model regions in the rendered image with the pre-collected logistics scene background image to obtain the package image, the true value information of each package model contained in the rendered image can be determined.

[0117] Optionally, the motion-completed package models can be colored and rendered from the perspective of a specified camera in a three-dimensional simulation space to obtain a second colored image. In the second colored image, the color value of the region where each package model is located is the same, and the color values ​​of the regions where different package models are located are different. Then, for each package model in the second colored image, the minimum bounding region of the color value corresponding to the package model can be determined from the colored image, and the coordinate information of each pixel in the minimum bounding region can be used as the ground truth information of the package.

[0118] In the above-described solution of this invention, package images can be automatically generated, improving the efficiency of acquiring package images in logistics scenarios and enabling the acquisition of richer information.

[0119] In one embodiment, to improve the realism of the acquired package images, after the motion simulation ends, the package models and package carrying platform models can be rendered from the perspective of a camera specified in the three-dimensional simulation space. Before obtaining the rendered image, lighting simulation can be performed in the three-dimensional simulation space; wherein, lighting simulation includes conventional lighting simulation and / or interference lighting simulation.

[0120] In simple terms, this involves simulating lighting in a logistics scenario, categorizing the lights into regular lights and interfering lights. The number of regular lights selected is L. num Place conventional lights at a fixed height of L z And in the horizontal direction, they are ±L x and ±L y The light is evenly distributed within a certain range, with a light intensity of E. normal According to probability P interface Randomly add interfering lights, selecting a quantity of 1 interfering light, with its location randomly within the range of regular lights, and a light intensity of E. interface .

[0121] In the above-described solution of this invention, package images can be automatically generated, which improves the efficiency of acquiring package images in logistics scenarios and enhances the authenticity of package images.

[0122] like Figure 5 As shown, an embodiment of the present invention provides an image generation method, including:

[0123] Step 1: Logistics model construction.

[0124] In this solution, the logistics model includes a platform model for carrying logistics packages and a package model. The logistics model is built within the GUI (Graphical User Interface) of the 3D engine software. The package carrying platform model is primarily achieved using geometric solid shapes through cutting, trimming, and splicing. After modeling, the model needs to be rendered with colors. The constructed package-carrying platform model structure is shown in the image. plane The logistics package model is divided into a rigid package model and a flexible package model. The rigid package model is composed of regular cubes; the flexible package model is a deformation based on the rigid package model. In this scheme, 100 models of different sizes are established for both the rigid and flexible package models, denoted as Model. hard With Model soft The established model also includes the material information of the wrapping model, including reflection parameters, refraction parameters, and fading parameters, reflecting the shape of the wrapping when illuminated. The rigid wrapping model contains two material information items: one is the surface texture material of the wrapping. face_hard One is a single-sided textured material. bill The flexible wrapping model includes material information, which is the surface texture material of the wrapping. face_soft .

[0125] Step 2: Image material collection.

[0126] When rendering package images in a logistics scene, a surface needs to be attached to the package model. Each surface image, i.e., an image carrying the package texture, can be bound to the material in Step 1. There are two types of surface images for rigid package models: one is a realistic surface image. face_hard Another type is a shipping label image. bill The surface map of the flexible wrapping model has only one image. uv_soft Image of a realistic flexible wrapping model surface. face_soft The unfolded diagram. Surface diagrams of the rigid-wrapped model and the flexible-wrapped model are shown below. Figure 5 As shown.

[0127] In addition to surface images, this step also requires images of the actual logistics scene. realCollecting real images of logistics scenarios refers to images taken by real cameras in logistics scenarios, including the logistics package carrying platform, the packages, and the background.

[0128] Step 3: Select the logistics package.

[0129] For a single simulation of a logistics scenario, it is necessary to select an appropriate number of packages for simulation.

[0130] When selecting packages, a combination of rigid and flexible package models is used. Based on the package density of different logistics scenarios, a package number between the minimum required number P and the required number of packages is selected. min With the maximum number of packages P max Random value P between num One package.

[0131] After selecting a logistics package, initialize its pose. The initialization method is to select horizontal and vertical ranges on the plane of the package-bearing platform model built in Step 1, denoted as ±Loc respectively. x With ±Loc y Select the height range of the package [0, Loc] above this horizontal range. z The package positions selected in Step 3 are randomly initialized in the above three-dimensional space to minimize collisions between packages, and the packages are randomly rotated.

[0132] Step 4: Wrap texture simulation.

[0133] For each package selected in Step 3, a surface image is extracted. Specifically, for each rigid package model, a single-sided surface image is randomly selected. face_hard Bind it to the texture material; then select a single-sided surface image. bill By varying the scale within a certain range, randomly rotating the image, and pasting it onto a single-sided surface image, a composite image (Image) is obtained. uv_hard And it is bound to the material of the faceplate. For each flexible wrapping model, a texture image of the flexible wrapping model is randomly selected. uv_soft It is bound to the texture material of the package.

[0134] Step 5: Package physics simulation.

[0135] The purpose of the physics simulation is to change the pose of all the packages selected in Step 3.

[0136] In this step, the physics simulation function in the 3D engine is used to add gravity simulation to all packages, causing all package models to fall and collide under the influence of gravity simulation. A fixed time length T is set for the physical effects on the packages.phy During this period, all packages are free to undergo pose changes in the physical simulation. At T phy After the time limit expires, packages in valid poses need to be filtered out, and those not in valid poses need to be deleted. In this step, a package is defined as having its center position on the horizontal plane not within ±Loc of Step 3. x and ±Loc y Packages within the range, and those not in the vertical direction [0, Loc z Packages within the specified range are considered illegal. In this step, all illegal packages will be deleted, and all legitimate packages will be retained.

[0137] Step 6: Camera pose simulation.

[0138] The camera setup rules for simulating a logistics scenario involve setting up a stereo camera in the 3D engine used. The camera position is set above the package feeding platform, and its fixed 3D coordinates are represented as (C...). x C y C z After setting the camera position, the camera's attitude needs to be adjusted. Here, adjusting the camera attitude means rotating the camera angle. In the camera pose settings, the camera angle rotation is also three-dimensional, with rotation ranges set in the x, y, and z directions. The final camera rotation angle is a random value within this rotation range. (Set as (R...) x ,R y ,R z ).

[0139] In this step, it is also necessary to set the camera parameters, including the camera's focal length in millimeters, the distance between the two eyes, the camera's field of view, the camera's minimum sensing distance, the camera's maximum sensing distance, and the scale of the image captured by the camera.

[0140] Step 7: Lighting simulation.

[0141] The lighting in a logistics scenario is simulated, with the lights divided into regular lights and interfering lights. The number of regular lights, L, is selected. num Place conventional lights at a fixed height of L z And in the horizontal direction, they are ±L x and ±L y The light is evenly distributed within a certain range, with a light intensity of E. normal According to probability P interface Randomly add interfering lights, selecting a quantity of 1 interfering light, with its location randomly within the range of regular lights, and a light intensity of E. interface .

[0142] Step 8: Image rendering.

[0143] Image rendering is divided into RGB image rendering, depth map rendering, and normal plane map rendering. First, set the world background for rendering to Color. bg Following the camera's single-image scale set in Step 6, the three types of images are rendered using the 3D engine's rendering interface to obtain the RGB image. rgb Depth map Image depth Image of the plan view normal .

[0144] Step 9: Image truth reconstruction.

[0145] Image truth reconstruction requires first rendering an instance segmentation map, and then performing truth reconstruction based on the rendered map.

[0146] Instance segmentation rendering. Using the coloring interface in the 3D engine, all existing packages and package feeding platforms are recolored. Each package is individually colored with a different color, denoted as Color. ins_1 Color ins_2 , ...Color ins_i And the packaging station was dyed Color. bg And render the instance segmentation image. ins In this image, the different packages are different colors, while the packaging station and the rest of the background are of the same color. bg .

[0147] Truth reconstruction. The Image rendered in this step... ins For the color in the image ins The truth reconstruction is performed on the region (each package corresponds to a different color) for instance segmentation. The truth form of instance segmentation is the bounding coordinates of each package. Specifically, the analysis is performed on the regions colored for different packages, and for the same color... ins_i In each region of the image, calculate the minimum set of bounding points. Finally, obtain a total of i bounding point sets for each region, which are considered the ground truth regions for instance segmentation. Record the ground truth of instance segmentation in the ground truth document File. gt middle.

[0148] Step 10: Background image blending.

[0149] Background image blending requires first rendering an image segmentation map, and then performing background blending on the rendered map.

[0150] Image segmentation and rendering. Using the coloring interface in the 3D engine, all existing packages and supply stations are colored, simultaneously coloring all packages and supply stations with Color. seg And render the image segmentation map Imageseg In this image, the color of the package and the feeding platform is Color. seg The background color for the rest is Color. bg .

[0151] For background fusion, a background image is randomly selected from the real images collected in Step 2 as the background image for image generation. real The fusion process requires the use of image segmentation maps. seg First, set the Image rgb With Image real Convert all to Image seg Same scale, then Image seg As a mask, filtering the real image rgb Image with background real In Image rgb Color in the upper mask seg The area in Image real Preserve Mask Color bg The two regions are then overlaid to obtain a fused simulation image. fusion And save the Image fusion .

[0152] Step 11: Iterative generation.

[0153] Repeat Steps 3 through 10 to generate different groups of diverse image data.

[0154] In the above-described scheme of this invention, through model building, physical simulation, texture simulation, and lighting simulation, a logistics package image that closely resembles a real logistics scene is rendered, and depth maps, normal plane maps, and instance segmentation labels are automatically generated. This simulation of logistics scenes effectively reduces the workload of sample collection and manual creation of instance segmentation labels, depth map and normal plane map annotations in deep learning.

[0155] Corresponding to the image generation method provided in the above embodiments of the present invention, such as Figure 6 As shown, this embodiment of the invention also provides an image generation apparatus, the apparatus comprising:

[0156] Model acquisition module 601 is used to acquire a specified number of package models;

[0157] The motion simulation module 602 is used to simulate the motion process of the specified number of package models on a preset package carrying platform model in a three-dimensional simulation space.

[0158] The image rendering module 603 is used to render the package models and the package carrying platform model after the motion simulation ends, from the perspective of a specified camera in the three-dimensional simulation space, to obtain a rendered image.

[0159] The image fusion module 604 is used to fuse the model region in the rendered image with the pre-collected logistics scene background image to obtain a package image, wherein the model region is the area where the package model and the package carrying platform model are located.

[0160] Optionally, the motion simulation module is specifically used to determine the initial pose of the specified number of package models within the designated area of ​​the package carrying platform model; for each package model, on the package carrying platform model, the initial pose of the package model is used as the starting pose of the motion, and motion simulation is performed on the package model.

[0161] Optionally, the motion simulation module is specifically used to perform a gravity simulation on the package carrying platform model for a specified duration, using the initial pose of the package model as the starting pose of the motion.

[0162] Optionally, the device further includes:

[0163] The camera setting module is used to set the camera at a specified position in the three-dimensional simulation space according to preset camera parameters before rendering each package model and the package carrying platform model after the motion is completed by the image rendering module, specifying the camera's perspective in the three-dimensional simulation space; adjusting the posture of the set camera; and using the posture-adjusted camera as the designated camera.

[0164] Optionally, the camera setting module is specifically used to adjust the posture of the set camera to a specified posture; or, to randomly adjust the posture of the set camera.

[0165] Optionally, the model acquisition module is specifically used to acquire package density information corresponding to the target logistics scenario; based on the package density information, determine the number of candidate package models as a specified number; and select the specified number of package models from the preset package models.

[0166] Optionally, the package density information indicates the maximum and / or minimum number of packages on the package carrying platform within the target logistics scenario; the specified number is less than or equal to the maximum number of packages, and / or greater than or equal to the minimum number of packages.

[0167] Optionally, the device further includes:

[0168] The texture bonding module is used to select a package texture for each package model from a preset set of package textures after the motion simulation module has obtained a specified number of package models and before simulating the motion process of the specified number of package models on a preset package carrying platform model in the three-dimensional simulation space, and to perform texture bonding on the package model using the selected package texture.

[0169] Optionally, the specified number of wrapping models include: rigid wrapping models; each wrapping texture includes a rigid wrapping surface texture and a single-sided texture;

[0170] The texture bonding module is specifically used to select, from the various rigid surface textures and single-sided textures, the rigid surface texture and the single-sided texture for the rigid surface model; to bond the selected rigid surface texture to the model surface of the rigid surface model; and to paste the selected single-sided texture onto the model surface on which the selected rigid surface texture has been bonded.

[0171] Optionally, the specified number of wrapping models include: flexible wrapping models; each wrapping texture includes a flexible wrapping surface texture;

[0172] The texture bonding module is specifically used to select a flexible wrapping surface texture for the flexible wrapping model from each flexible wrapping surface texture; and to bond the selected flexible wrapping surface texture to the model surface of the flexible wrapping model.

[0173] Optionally, the device further includes:

[0174] The truth information determination module is used to determine the truth information of each wrapping model contained in the rendered image.

[0175] Optionally, the truth information determination module is specifically used to perform color rendering on each package model after the motion is completed from the perspective of a specified camera in the three-dimensional simulation space to obtain a first color image, wherein the color value of the region where each package model is located in the second color image is the same, and the color values ​​of the regions where different package models are located are different; for each package model in the second color image, the minimum bounding region of the color value corresponding to the package model is determined from the color image, and the coordinate information of each pixel in the minimum bounding region is used as the truth information of the package.

[0176] Optionally, the device further includes:

[0177] The model region determination module is used to perform color rendering on each package model after motion is completed from the perspective of a specified camera in the three-dimensional simulation space before the image fusion module performs the fusion of the model region in the rendered image with the pre-acquired logistics scene background image to obtain the package image, in order to obtain a second color image. In the first color image, the color value of the area where the package model and the package carrying platform model are located is a first color value, and the color value of the background area in the first color image is a second color value. The module determines the smallest bounding region in the first color image as the model region.

[0178] Optionally, the image fusion module is specifically used to generate a region mask for the model region based on the model region in the first stained image; in the rendered image, the image region corresponding to the region mask is retained as the first image region, and the image region corresponding to the region mask is removed from the pre-acquired logistics scene background image to obtain the second image region; the first image region and the second image region are merged to obtain the package image.

[0179] Optionally, the device further includes:

[0180] The lighting simulation module is used to perform lighting simulation in the three-dimensional simulation space before the image rendering module executes the step of rendering each package model and the package carrying platform model from the perspective of a specified camera in the three-dimensional simulation space after the motion simulation ends, and before obtaining the rendered image; wherein, the lighting simulation includes conventional lighting simulation and / or interference lighting simulation.

[0181] In the above-described solution of this invention, since the package model can be simulated on the package carrying platform model to obtain a rendered image, and then combined with the logistics scene background image to obtain a package image, the package image can be automatically generated, thereby improving the efficiency of obtaining package images in the logistics scene.

[0182] This invention also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0183] Memory 703 is used to store computer programs;

[0184] When processor 701 executes a program stored in memory 703, it performs the following steps:

[0185] Get a specified number of package models;

[0186] In a three-dimensional simulation space, the motion process of the specified number of package models on a preset package carrying platform model is simulated.

[0187] After the simulated motion ends, the rendering images of each package model and the package carrying platform model are obtained from the perspective of a specified camera in the three-dimensional simulation space.

[0188] The model region in the rendered image is fused with a pre-collected logistics scene background image to obtain a package image, wherein the model region is the area where the package model and the package carrying platform model are located.

[0189] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0190] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0191] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0192] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0193] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image generation methods.

[0194] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image generation methods described in the above embodiments.

[0195] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0196] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0197] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer storage media, and computer program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0198] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An image generation method, characterized in that, The method includes: Get a specified number of package models; In a three-dimensional simulation space, the motion process of the specified number of package models on a preset package carrying platform model is simulated. After the simulated motion ends, the rendering images of each package model and the package carrying platform model are obtained from the perspective of a specified camera in the three-dimensional simulation space. The model region in the rendered image is fused with a pre-collected logistics scene background image to obtain a package image, wherein the model region is the area where the package model and the package carrying platform model are located; Before fusing the model region in the rendered image with the pre-acquired logistics scene background image to obtain the package image, the method further includes: Using the perspective of the designated camera in the three-dimensional simulation space, each package model after the motion is completed is colored and rendered to obtain a first colored image, wherein the color value of the area where the package model and the package carrying platform model are located in the first colored image is a first color value, and the color value of the background area in the first colored image is a second color value. The smallest bounding region containing the first color value in the first stained image is determined as the model region.

2. The method according to claim 1, characterized in that, The simulation of the movement of the specified number of package models on a preset package carrying platform model includes: Determine the initial pose of the specified number of package models within the region represented by the package carrying platform model; For each package model, on the package carrying platform model, the initial pose of the package model is used as the starting pose for motion simulation of the package model.

3. The method according to claim 2, characterized in that, The motion simulation of the package model on the package-carrying platform model, using the initial pose of the package model as the starting pose of the motion, includes: On the package-carrying platform model, the initial pose of the package model is used as the starting pose for motion, and a gravity simulation of the package model is performed for a specified duration.

4. The method according to claim 1, characterized in that, The step of fusing the model region in the rendered image with a pre-collected logistics scene background image to obtain a package image includes: Based on the model region in the first coloring map, a region mask for the model region is generated; In the rendered image, the image region corresponding to the region mask is retained as the first image region, and the image region corresponding to the region mask is removed from the pre-collected logistics scene background image to obtain the second image region; The first image region and the second image region are merged to obtain the wrapped image.

5. The method according to claim 1, characterized in that, After acquiring a specified number of package models, and before simulating the motion of the specified number of package models on a preset package-carrying platform model in a three-dimensional simulation space, the method further includes: For each package model, select a package texture from the preset package textures, and then use the selected package texture to perform texture fitting on the package model.

6. The method according to claim 5, characterized in that, The specified number of package models include: rigid package models; each package texture includes a rigid package surface texture and a single-sided texture; The step of selecting a wrapping texture for the wrapping model from a preset set of wrapping textures, and then using the selected wrapping texture to perform texture bonding on the wrapping model, includes: Select the hard-wrapped surface texture and single-face texture for the hard-wrapped model from the various hard-wrapped surface textures and single-face textures. Apply the selected hard-covered surface texture to the surface of the hard-covered model. On the model surface that already has the selected hard-surface texture applied, paste the selected single-sided texture.

7. The method according to claim 5, characterized in that, The specified number of package models include: flexible package models; each package texture includes a flexible package surface texture; The step of selecting a wrapping texture for the wrapping model from a preset set of wrapping textures, and then using the selected wrapping texture to perform texture bonding on the wrapping model, includes: Select the flexible wrapping surface texture for this flexible wrapping model from the various flexible wrapping surface textures; Apply the selected flexible wrapping surface texture to the surface of the flexible wrapping model.

8. The method according to claim 1, characterized in that, Before rendering the package models and the package-carrying platform model after the motion is completed, using the viewpoint of a camera specified in the three-dimensional simulation space, to obtain a rendered image, the method further includes: Set up the camera at a specified location in the three-dimensional simulation space according to the preset camera parameters; Adjust the orientation of the set camera and use the camera with the adjusted orientation as the designated camera.

9. The method according to claim 8, characterized in that, The adjustment of the camera's orientation includes: Adjust the camera's orientation to the specified orientation; or, randomly adjust the camera's orientation.

10. The method according to claim 1, characterized in that, The step of obtaining a specified number of package models includes: Obtain package density information corresponding to the target logistics scenario; Based on the package density information, the number of candidate package models is determined as the specified number; Select the specified number of package models from the preset package models.

11. The method according to claim 10, characterized in that, The parcel density information indicates the maximum and / or minimum number of parcels on the parcel carrying platform within the target logistics scenario. The specified quantity is less than or equal to the maximum number of packages, and / or greater than or equal to the minimum number of packages.

12. The method according to claim 1, characterized in that, Before fusing the model region in the rendered image with the pre-acquired logistics scene background image to obtain the package image, the method further includes: The rendered image is determined to contain truth information about each package model.

13. The method according to claim 12, characterized in that, The step of determining that the rendered image contains the wrapping ground truth information of each wrapping model includes: Using the perspective of the specified camera in the three-dimensional simulation space, each package model after the motion is completed is colored and rendered to obtain a second colored image. In the second colored image, the color value of the area where each package model is located is the same, and the color value of the area where different package models are located is different. For each enclosed model in the stained image, the minimum enclosing region of the color value corresponding to the enclosed model is determined from the second stained image, and the coordinate information of each pixel within the minimum enclosing region is used as the ground truth information of the enclosed model.

14. The method according to claim 1, characterized in that, Before rendering the package models and the package-carrying platform model from the perspective of a specified camera in the three-dimensional simulation space after the motion simulation ends, and obtaining the rendered image, the method further includes: Lighting simulation is performed in the three-dimensional simulation space; wherein, the lighting simulation includes conventional lighting simulation and / or interference lighting simulation.

15. An image generation apparatus, characterized in that, The device includes: The model acquisition module is used to acquire a specified number of package models; The motion simulation module is used to simulate the motion process of the specified number of package models on a preset package carrying platform model in a three-dimensional simulation space. The image rendering module is used to render the package models and the package carrying platform model after the motion simulation ends, from the perspective of a specified camera in the three-dimensional simulation space, to obtain a rendered image. The image fusion module is used to fuse the model region in the rendered image with a pre-collected logistics scene background image to obtain a package image, wherein the model region is the area where the package model and the package carrying platform model are located; The device further includes: The model region determination module is used to perform color rendering on each package model after motion is completed from the perspective of the specified camera in the three-dimensional simulation space before the image fusion module performs the fusion of the model region in the rendered image with the pre-acquired logistics scene background image to obtain the package image, in order to obtain a first color image. In the first color image, the color value of the area where the package model and the package carrying platform model are located is a first color value, and the color value of the background area in the first color image is a second color value. The module determines the smallest bounding area where the first color value is located in the first color image as the model region.

16. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-14.

Images