Image velocity field data set construction method and device based on digital twinning

Abstract

The invention provides an image velocity field data set construction method and device based on digital twinning. The method comprises the following steps: building a motion scene of an instance; setting instance motion information, and determining a position vector, a linear velocity field and an angular velocity field of an instance; step 3, generating an instance mask, and obtaining an instance color mask diagram and a corresponding position vector diagram, a linear velocity vector diagram and an angular velocity vector diagram; step 4, image rendering: performing image synthesis based on the position vector diagram, the linear velocity field diagram, the angular velocity field diagram and the instance color mask diagram of the object by using a rendering engine to obtain a visible light image of the instance; 5, setting the number of samples of the data set, and taking the number of samples as the number of cycles for cyclically executing the steps 1-4; and obtaining an instance image sample based on the visible light image of the instance in each cycle. According to the method, the accuracy of marking the potential vector field and the velocity field is improved, the adaptability of the data set to different image acquisition devices is improved, and the cost and difficulty of marking the velocity field are reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for constructing an image velocity field data set based on digital twins. Background Art

[0002] Velocity estimation is one of the main technologies studied in the field of computer vision. This technology aims to obtain the motion state of different object instances in the visual input, including the current position, velocity, angular velocity, etc. This technology is widely used in transportation, industrial production lines, competitive sports, atmospheric science and other fields, and can play a vital role in the decision-making of unmanned driving systems, the health status monitoring of assembly lines, the analysis of athlete performance, and the prediction of atmospheric motion laws.

[0003] Velocity fields are generally used in fields such as fluid mechanics to determine the motion state of certain particles or particles in the fluid. Due to the fluidity of the fluid, at the microscopic level, the particles that make up the fluid can coexist with huge differences at the same time. This phenomenon is also of reference significance in general velocity estimation tasks, especially for the motion state of multi-degree-of-freedom targets, flexible targets, and cluster targets. The velocity field can predict the pixel-level velocity of each instance in the image, and analyze its motion state based on linear velocity and angular velocity from the whole to the local. Summary of the invention

[0004] The present invention provides a method and device for constructing an image velocity field data set based on digital twins. The method is based on the concept of digital twins, completes scene construction through 3D modeling simulation, and then uses a rendering engine for image synthesis. On the other hand, through the instance motion state setting, four RGB three-channel images generated by direct calculation are added to each sample through fixed-point calculation, which are used to represent an instance mask image of a unique color in a corresponding color table, as well as a position vector map, velocity field map, and angular velocity field map with three dimensions.

[0005] The technical solution of the present invention is: a method for constructing an image velocity field dataset based on digital twins, which comprises the following steps:

[0006] Step 1: Use 3D modeling simulation to complete the construction of the motion scene of the instance; the object motion scene includes: the object model of the instance, the screen space, and the virtual camera;

[0007] Step 2: Set the motion information of the instance, which includes linear velocity and angular velocity. The screen space emits a ray along the line of sight of the virtual camera, records the intersection of the ray and the first triangle face of the object it hits, and determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information;

[0008] Step 3: Instance mask generation: the screen space will emit rays along the line of sight of the virtual camera, record the intersection of the ray and the first instance triangle it hits, take the instance to which the triangle belongs as the instance corresponding to the point, set the colors of all instances in the scene, where each instance corresponds to a color in the color table, and obtain the instance color mask map and the corresponding position vector map, linear velocity vector map, and angular velocity vector map;

[0009] Step 4, rendering the image, using a rendering engine to perform image synthesis based on the object's position vector map, linear velocity field map, angular velocity field map, and instance color mask map to obtain a visible light image of the instance;

[0010] Step 5: Set the number of samples in the data set, and use the number of samples as the number of loops to execute steps 1 to 4; obtain an instance image sample based on the visible light image of the instance in each loop; exit the loop when the expected number of instance image samples is obtained.

[0011] In one embodiment, the position vector, linear velocity field and angular velocity field of the instance are determined according to the position information of the ray at the intersection and the screen space information, and the specific method is:

[0012] The method for determining the position vector is as follows: using a spherical coordinate system in the screen space where the instance is located, taking the direction of the virtual camera and the vertical axis of the screen space as the θ plane, the direction of the camera and the horizontal axis of the screen space as the φ plane, and the line connecting the center point of the camera and the target point as the distance ρ, to determine the position vector of the instance; in terms of time, the time difference between two frames is used as the time unit;

[0013] The method for determining the linear velocity field is: linear velocity v,

[0014] Among them, Δθ, Δφ, Δρ are the differences between the three components of the position vector between two adjacent frames on the three planes θ, φ, ρ of the spherical coordinate system; Δt is the time difference between two frames. Taking Δt as the time unit, Δt=1, and the linear velocity is obtained as:

[0015] v = Δθ + Δφ + Δρ;

[0016] The method for determining the angular velocity field is as follows: a left-handed system is defined using a rectangular space, with the rightward direction of the screen space as the positive direction of the x-axis, the upward direction of the screen space as the positive direction of the y-axis, and the direction of the camera's line of sight as the z-axis, with the camera's line of sight perpendicular to the screen space and inward; in terms of time, Δt is taken as the time unit, then Δt = 1, and the angular velocity is:

[0017] ω=Δα x +Δα y +Δα z ,

[0018] Among them, Δαx, Δαy, and Δαz are the differences of the three components αx, αy, and αz of the attitude angle of two adjacent frames on the x-axis, y-axis, and z-axis at a certain point.

[0019] In one embodiment, the method for setting the colors of all instances in the scene is: setting a color for each instance according to a color table, wherein each instance corresponds to a color in the color table.

[0020] In one embodiment, it further includes: the example image samples include: a visible light image, a position vector image, a linear velocity vector image and an angular velocity vector image of the example; and their corresponding images in the three channels of R, G and B in the RGB color space.

[0021] A device for constructing an image velocity field data set based on digital twins, comprising:

[0022] The scene building module has the following functions: using 3D modeling simulation to complete the motion scene building of the instance; the object motion scene includes: the object model of the instance, screen space, and virtual camera;

[0023] The motion information setting module has the following functions: the motion information includes linear velocity and angular velocity. The screen space emits a ray along the line of sight of the virtual camera, records the intersection of the ray and the first triangle face of the object it hits, and determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information;

[0024] The instance mask generation module has the following functions: the screen space emits rays along the line of sight of the virtual camera, records the intersection of the ray and the first instance triangle it hits, takes the instance to which the triangle belongs as the instance corresponding to the point, sets the colors of all instances in the scene, where each instance corresponds to a color in the color table, and obtains the instance color mask map and the corresponding position vector map, linear velocity vector map, and angular velocity vector map;

[0025] The rendering engine has the following functions: to synthesize images based on the position vector map, linear velocity field map, angular velocity field map, and instance color mask map of the object to obtain a visible light image of the instance;

[0026] The sample batch generation module has the following functions: setting the number of samples in the data set, using the number of samples as the number of cycles for the scene building module, motion information setting module, instance mask generation module, and rendering engine to run in sequence; obtaining an instance image sample based on the visible light image of the instance in each cycle; and exiting the cycle when the expected number of instance image samples is obtained.

[0027] In one embodiment, the motion information setting module determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information, and the specific method is:

[0028] The method for determining the position vector is as follows: using a spherical coordinate system in the screen space where the instance is located, taking the direction of the virtual camera and the vertical axis of the screen space as the θ plane, the direction of the camera and the horizontal axis of the screen space as the φ plane, and the line connecting the center point of the camera and the target point as the distance ρ, to determine the position vector of the instance; in terms of time, the time difference between two frames is used as the time unit;

[0029] The method for determining the linear velocity field is: linear velocity v,

[0030] Among them, Δθ, Δφ, Δρ are the differences between the three components of the position vector between two adjacent frames on the three planes θ, φ, ρ of the spherical coordinate system; Δt is the time difference between two frames. Taking Δt as the time unit, Δt=1, and the linear velocity is obtained as:

[0031] v = Δθ + Δφ + Δρ;

[0032] The method for determining the angular velocity field is as follows: using a rectangular space definition, a left-handed system, with the right direction of the screen space as the positive direction of the x-axis, the upward direction of the screen space as the positive direction of the y-axis, and the direction of the camera's line of sight as the z-axis, with the camera's line of sight perpendicular to the screen space and inward; in terms of time, taking Δt as the time unit, then Δt = 1, the angular velocity is:

[0033] ω=Δα x +Δα y +Δα z ,

[0034] Among them, Δαx, Δαy, and Δαz are the differences of the three components αx, αy, and αz of the attitude angle of two adjacent frames on the x-axis, y-axis, and z-axis at a certain point.

[0035] In one embodiment, the method for setting the colors of all instances in the scene is: setting a color for each instance according to a color table, wherein each instance corresponds to a color in the color table.

[0036] In one embodiment, the example image samples include: a visible light image, a position vector image, a linear velocity vector image, and an angular velocity vector image of the example; and their corresponding images in three channels of R, G, and B in an RGB color space.

[0037] The advantages of the present invention compared with the prior art are:

[0038] The present invention improves the accuracy of position and velocity field annotation. The model and animation information used in the scene construction of the present invention are all manually set, which can directly and accurately calculate the position, linear velocity, angular velocity, etc. of each vertex and spatial pixel of the target relative to the camera.

[0039] The present invention improves the adaptability of data sets to different image acquisition devices. The present invention uses frames and spherical coordinates as time and space units, respectively, to avoid system differences caused by sensors to the greatest extent. By defining sensor information such as field of view angle, focal length, sampling rate, etc., it can be converted into standard angular velocity and linear velocity.

[0040] The present invention reduces the cost and difficulty of velocity field annotation. The pixel-level velocity field annotation of the present invention completely relies on program calculation, can be fully automated, and avoids the huge labor cost and time cost of annotating the velocity field frame by frame, instance by instance, or even pixel by pixel through the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the method for constructing an image velocity field data set of the present invention. DETAILED DESCRIPTION

[0042] The present invention proposes a method and device for constructing an image velocity field dataset based on digital twins. Figure 1 , the method of the present invention comprises the following steps:

[0043] Step 1, scene initialization. Clear the previous scene and import all models, textures, light sources, particle emitters, physical fields, virtual cameras, etc. used to build this scene as scene assets into the scene. Here, the model refers to the "3D model", which is a data structure used to represent a three-dimensional object and is responsible for providing shape, texture, normal and other information to the 3D modeling software. Textures are used to further improve the authenticity of the model and define information other than the vertices on each triangular face of the model, including basic color, roughness, metalness, normal direction, etc. Particle emitters are used to generate particles in 3D modeling software. These particles are usually used to simulate various special effects, including flames, smoke, and fluids. The physical field is a tool in 3D modeling software that is used to control the physical simulation of specified objects and particles in a scene of the 3D software. It can interact with objects in the form of neutral, magnetic, wind field, eddy current, orbital constraints, etc. Based on the principle of domain randomization, the position, posture, size and other parameters of these instances are randomly set, and finally they form a fairly general scene. This step is the fundamental prerequisite to ensure that the rendering results are consistent with reality in terms of feature distribution. In addition, heterogeneous and randomized scene design is also the key to the rich variety and diverse features of the data set.

[0044] Step 2, set the object motion information. The motion information includes linear velocity and angular velocity. The screen space emits rays along the line of sight of the virtual camera, records the intersection of the ray and the first triangle of the object it hits, and determines the position vector, linear velocity field and angular velocity field of the instance based on the position information of the ray at the intersection and the screen space information. The object motion information here includes linear velocity and angular velocity, both driven by key frames and animation curves. In this step, the screen space will emit rays along the line of sight of the virtual camera, and the intersection with the first triangle hit will be used as the three-dimensional velocity vector of the pixel. By interpolating the motion state of the three vertices constituting the triangle at this time, the instance to which the intersection belongs and the three-dimensional motion information of the intersection, including the position vector, angular velocity and linear velocity, can be obtained.

[0045] In order to avoid the influence of image acquisition device parameters on linear velocity estimation, the position vector field and velocity field used in the present invention are defined as follows. In space, a spherical coordinate system is used, with the camera direction and the vertical axis of the screen space as the θ plane, the camera direction and the horizontal axis of the screen space as the φ plane, and the line connecting the camera center point and the target point as the distance ρ. In time, a frame is used as the time unit.

[0046] The linear velocity is then defined as:

[0047]

[0048] v consists of three parts, namely Δθ / Δt, Δφ / Δt, Δρ / Δt, Δθ, Δφ, Δρ are the three components of the position vector between two adjacent frames, and the difference between θ, φ, and ρ. Taking the frame as the time unit, then Δt = 1, the linear velocity is:

[0049] v=Δθ+Δφ+Δρ

[0050] Angular velocity is not affected by camera parameters and is defined in rectangular space, with the left-handed system, with the right direction of the screen space as the positive direction of the x-axis, the upward direction of the screen space as the positive direction of the y-axis, and the camera's line of sight (vertical to the screen space) as the z-axis. In terms of time, the frame is still used as the time unit. So the angular velocity is defined as:

[0051] ω=Δα x +Δα y +Δα z

[0052] Among them, Δαx, Δαy, and Δαz are the differences between the three components of the attitude angle αx, αy, and αz of two adjacent frames at a certain point.

[0053] On this basis, the problem is redefined. The behavior of the target network finally trained by the present invention is actually to determine the parameters of a tensor transformation, which is an integer defined in the range [0,255]. Mapping. The ultimate function of this neural network is to perform pixel-level instance masking, as well as reconstruction of the position vector field, velocity field, and angular velocity field based on the given RGB three-channel visible light image input.

[0054] Step 3: Generate instance mask. This step implements pixel-level instance segmentation. The implementation method is similar to step 2. The screen space will emit rays along the line of sight of the virtual camera, and the object to which the first triangle hit belongs is the instance corresponding to the pixel. All independent instance objects in the scene will correspond to a color in the color table to maintain temporal continuity. In other words, masks with the same color appearing in two adjacent frames must correspond to the same instance.

[0055] Step 4: Render the image. Using the rendering engine and the principles of computer graphics, the image is synthesized based on the position vector map, linear velocity field map, angular velocity field map, and instance color mask map of the object to obtain a visible light image of the instance.

[0056] Step 5, batch sample generation. Set the expected number of data set samples, use the number of samples as the number of loops to loop through steps 1 to 4, and execute steps 1 to 4. Each loop obtains an instance image sample based on the visible light image of the instance; exit the loop until the expected number of instance image samples is obtained. The instance image samples include: the visible light image, position vector image, linear velocity vector image, and angular velocity vector image of the instance; and their corresponding images of the three channels of R, G, and B in the RGB color space.

[0057] Accordingly, the present invention also proposes a device for constructing an image velocity field dataset based on digital twins, which comprises:

[0058] The scene building module has the following functions: using 3D modeling simulation to complete the motion scene building of the instance; the object motion scene includes: the object model of the instance, screen space, and virtual camera;

[0059] The motion information setting module has the following functions: the motion information includes linear velocity and angular velocity. The screen space emits a ray along the line of sight of the virtual camera, records the intersection of the ray and the first triangle face of the object it hits, and determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information;

[0060] The instance mask generation module has the following functions: the screen space emits rays along the line of sight of the virtual camera, records the intersection of the ray and the first instance triangle it hits, takes the instance to which the triangle belongs as the instance corresponding to the point, sets the colors of all instances in the scene, where each instance corresponds to a color in the color table, and obtains the instance color mask map and the corresponding position vector map, linear velocity vector map, and angular velocity vector map;

[0061] The rendering engine has the following functions: to synthesize images based on the position vector map, linear velocity field map, angular velocity field map, and instance color mask map of the object to obtain a visible light image of the instance;

[0062] The sample batch generation module has the following functions: setting the number of samples in the data set, using the number of samples as the number of cycles for the scene building module, motion information setting module, instance mask generation module, and rendering engine to run in sequence; obtaining an instance image sample based on the visible light image of the instance in each cycle; and exiting the cycle when the expected number of instance image samples is obtained.

[0063] The motion information setting module determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information, and the specific method is as follows:

[0064] The method for determining the position vector is as follows: using a spherical coordinate system in the screen space where the instance is located, with the virtual camera direction and the vertical axis of the screen space as the θ plane, the camera direction and the horizontal axis of the screen space as the φ plane, and the line connecting the camera center point and the target point as the distance ρ to determine the instance's position vector; in terms of time, using frames as the time unit

[0065] The method for determining the linear velocity field is: linear velocity v,

[0066] Among them, Δθ, Δφ, Δρ are the differences between the three components of the position vector between two adjacent frames on the three planes θ, φ, ρ of the spherical coordinate system; Δt is the time difference between two frames. Taking Δt as the time unit, Δt=1, and the linear velocity is obtained as:

[0067] v = Δθ + Δφ + Δρ;

[0068] The method for determining the angular velocity field is as follows: using a rectangular space definition, a left-handed system, with the right direction of the screen space as the positive direction of the x-axis, the upward direction of the screen space as the positive direction of the y-axis, and the direction of the camera's line of sight as the z-axis, with the camera's line of sight perpendicular to the screen space and inward; in terms of time, taking Δt as the time unit, then Δt = 1, the angular velocity is:

[0069] ω=Δα x +Δα y +Δα z ,

[0070] Among them, Δαx, Δαy, and Δαz are the differences of the three components αx, αy, and αz of the attitude angle of two adjacent frames on the x-axis, y-axis, and z-axis at a certain point.

[0071] The method to set the color of all instances in the scene is to set a color for each instance according to the color table, where each instance corresponds to a color in the color table.

[0072] The example image samples include: a visible light image, a position vector image, a linear velocity vector image, and an angular velocity vector image of the example; and their corresponding images in three channels of R, G, and B in the RGB color space.

Claims

1. A method for constructing an image velocity field dataset based on digital twins, characterized in that: It includes the following steps: Step 1: Use 3D modeling simulation to complete the construction of the motion scene of the example; The object motion scene includes: the object model of the instance, the screen space, and the virtual camera; Step 2: Set the motion information of the instance, which includes linear velocity and angular velocity. The screen space emits a ray along the line of sight of the virtual camera, records the intersection of the ray and the first triangle face of the object it hits, and determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information. Step 3: Instance mask generation: the screen space will emit rays along the line of sight of the virtual camera, record the intersection of the ray and the first instance triangle it hits, take the instance to which the triangle belongs as the instance corresponding to the point, set the colors of all instances in the scene, where each instance corresponds to a color in the color table, and obtain the instance color mask map and the corresponding position vector map, linear velocity vector map, and angular velocity vector map; Step 4, rendering the image, using a rendering engine to synthesize the image based on the position vector map, linear velocity field map, angular velocity field map, and instance color mask map of the object to obtain a visible light image of the instance; Step 5: Set the number of samples in the data set, and use the number of samples as the number of loops to execute steps 1 to 4; obtain an instance image sample based on the visible light image of the instance in each loop; exit the loop when the expected number of instance image samples is obtained.

2. The method according to claim 1, characterized in that The position vector, linear velocity field and angular velocity field of the instance are determined according to the position information of the ray at the intersection and the screen space information. The specific method is: The method for determining the position vector is as follows: using a spherical coordinate system in the screen space where the instance is located, taking the direction of the virtual camera and the vertical axis of the screen space as the θ plane, the direction of the camera and the horizontal axis of the screen space as the φ plane, and the line connecting the center point of the camera and the target point as the distance ρ, to determine the position vector of the instance; in terms of time, the time difference between two frames is used as the time unit; The method for determining the linear velocity field is: linear velocity v, Among them, Δθ, Δφ, Δρ are the differences between the three components of the position vector between two adjacent frames on the three planes θ, φ, ρ of the spherical coordinate system; Δt is the time difference between two frames. Taking Δt as the time unit, Δt=1, and the linear velocity is obtained as: v = Δθ + Δφ + Δρ; The method for determining the angular velocity field is as follows: a left-handed system is defined using a rectangular space, with the rightward direction of the screen space as the positive direction of the x-axis, the upward direction of the screen space as the positive direction of the y-axis, and the direction of the camera's line of sight as the z-axis, with the camera's line of sight perpendicular to the screen space and inward; in terms of time, Δt is taken as the time unit, then Δt = 1, and the angular velocity is: ω=Da x +Da y +Da z , Among them, Δαx, Δαy, and Δαz are the differences of the three components αx, αy, and αz of the attitude angle of two adjacent frames on the x-axis, y-axis, and z-axis at a certain point.

3. The method according to claim 1, characterized in that The method of setting the colors of all instances in the scene is: setting a color for each instance according to the color table, where each instance corresponds to a color in the color table.

4. The method according to claim 1, characterized in that: It further includes: the example image samples include: a visible light image, a position vector image, a linear velocity vector image and an angular velocity vector image of the example; and their corresponding images in three channels of R, G and B in the RGB color space.

5. A device for constructing an image velocity field dataset based on digital twins, characterized in that: It includes: The scene building module has the following functions: using 3D modeling simulation to complete the construction of the instance's motion scene; The object motion scene includes: the object model of the instance, the screen space, and the virtual camera; The motion information setting module has the following functions: the motion information includes linear velocity and angular velocity. The screen space emits a ray along the line of sight of the virtual camera, records the intersection of the ray and the first triangle face of the object it hits, and determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information; The instance mask generation module has the following functions: the screen space emits rays along the line of sight of the virtual camera, records the intersection of the ray and the first instance triangle it hits, takes the instance to which the triangle belongs as the instance corresponding to the point, sets the colors of all instances in the scene, where each instance corresponds to a color in the color table, and obtains the instance color mask map and the corresponding position vector map, linear velocity vector map, and angular velocity vector map; The rendering engine has the following functions: to synthesize images based on the position vector map, linear velocity field map, angular velocity field map, and instance color mask map of the object to obtain a visible light image of the instance; The sample batch generation module has the following functions: setting the number of samples in the data set, using the number of samples as the number of cycles for the scene building module, motion information setting module, instance mask generation module, and rendering engine to run in sequence; obtaining an instance image sample based on the visible light image of the instance in each cycle; and exiting the cycle when the expected number of instance image samples is obtained.

6. The device according to claim 5, characterized in that The motion information setting module determines the position vector, linear velocity field and angular velocity field of the instance according to the position information of the ray at the intersection and the screen space information, and the specific method is as follows: The method for determining the position vector is as follows: using a spherical coordinate system in the screen space where the instance is located, taking the direction of the virtual camera and the vertical axis of the screen space as the θ plane, the direction of the camera and the horizontal axis of the screen space as the φ plane, and the line connecting the center point of the camera and the target point as the distance ρ, to determine the position vector of the instance; in terms of time, the time difference between two frames is used as the time unit; The method for determining the linear velocity field is: linear velocity v, Among them, Δθ, Δφ, Δρ are the differences between the three components of the position vector between two adjacent frames on the three planes θ, φ, ρ of the spherical coordinate system; Δt is the time difference between two frames. Taking Δt as the time unit, Δt=1, and the linear velocity is obtained as: v = Δθ + Δφ + Δρ; The method for determining the angular velocity field is as follows: using a rectangular space definition, a left-handed system, with the right direction of the screen space as the positive direction of the x-axis, the upward direction of the screen space as the positive direction of the y-axis, and the direction of the camera's line of sight as the z-axis, with the camera's line of sight perpendicular to the screen space and inward; in terms of time, taking Δt as the time unit, then Δt = 1, the angular velocity is: ω=Da x +Da y +Da z , Among them, Δαx, Δαy, and Δαz are the differences of the three components αx, αy, and αz of the attitude angle of two adjacent frames on the x-axis, y-axis, and z-axis at a certain point.

7. The device according to claim 5, characterized in that The method to set the color of all instances in the scene is to set a color for each instance according to the color table, where each instance corresponds to a color in the color table.

8. The device according to claim 5, characterized in that The example image samples include: a visible light image, a position vector image, a linear velocity vector image, and an angular velocity vector image of the example; and their corresponding images in three channels of R, G, and B in the RGB color space.

Images