A simulation method for visual sensors in Unity3D suitable for ROS system

Abstract

The present invention provides a simulation method of a visual sensor under Unity3D applicable to a ROS system. By means of Unity3D simulation, a visual sensor in a virtual environment is realized, and the sensor communicates with an actual ROS system, thereby providing a key technical method for a robot simulation test environment. The present invention simulates interference data such as noise that would appear in a real camera by adding selectable image noise, thereby improving the realism of the simulated camera. By processing and encoding RGB image data, the present invention generates ROS sensor data in an Image format, including complete unprocessed RGB image data, thereby realizing the RGB raw data simulation of an RGB camera in Unity3D. By compressing and encoding RGB image data, the present invention generates ROS sensor data in a CompressedImage format, thereby realizing the RGB compressed data simulation of an RGB camera in Unity3D.

Description

Technical Field

[0001] The present invention relates to a simulation method of a visual sensor under Unity3D applicable to a ROS system, which provides a key technical method for a robot simulation test environment. Background Art

[0002] As an open source robot operating system platform, ROS is widely used in the service robot industry with its active ecological community. When studying the robot's algorithm, it is often necessary to test its functions and performance in different environments. However, due to site and resource constraints, it is unlikely to provide such an arbitrary changing place at any time, so a virtual simulation environment is the best solution for robot algorithm evaluation.

[0003] The Chinese patent application with application number 201810021768.5 discloses a robot cross-platform communication protocol and control method. The invention patent application uses Unity3D, ROS-bridge and ROS system to implement a robot cross-platform communication protocol and control method. The cross-platform communication protocol stated in the above invention patent application only includes robot control instructions, which is insufficient in content and poor in applicability. Summary of the invention

[0004] The purpose of the present invention is to realize a visual sensor in a virtual environment by means of Unity3D simulation and communicate with an actual ROS system.

[0005] In order to achieve the above object, the technical solution of the present invention is to provide a simulation method of a visual sensor under Unity3D applicable to the ROS system, characterized in that it is used to obtain an RGB image or a depth image, wherein:

[0006] Obtaining an RGB image by the simulation method comprises the following steps:

[0007] Step 101: Use the Camera component in the Unity3D 3D game engine on the Windows platform to obtain the original visual field;

[0008] Step 102: Modify the resolution, horizontal field of view angle, vertical field of view angle, depth of field, refresh rate, and image quality of the Camera component based on the original visual domain, where:

[0009] Modify the resolution of the Camera component according to the resolution in the actual camera parameters. The resolution is expressed as image width W × height H;

[0010] The vertical field of view angle in the actual camera parameters requires modification of the vertical field of view angle HFOV;

[0011] The horizontal field of view angle WFOV is calculated according to the relationship between the resolution W×H, the vertical field of view angle HFOV, the horizontal field of view angle WFOV and the focal length f of the camera, and the horizontal field of view angle WFOV of the Camera component is modified according to the calculated horizontal field of view angle WFOV;

[0012] Modify the depth of field according to the nearest depth of field and the farthest depth of field in the actual camera parameters;

[0013] Modify the refresh rate according to the corresponding requirements of the refresh rate in the actual camera parameters;

[0014] Dynamically set the corresponding image compression ratio according to the bandwidth of the target user to adjust the image quality;

[0015] Step 103: Create a new canvas component, intercept the canvas component according to the resolution determined in step 102, obtain a canvas component consistent with the resolution, assign the obtained original visible field pixels to the canvas component by row and column, and return the corresponding canvas component, which is the generated RGB image;

[0016] Step 104: adding corresponding noise to the generated RGB image;

[0017] Step 105: According to the Linux platform ROS system communication format, the RGB image obtained in the previous step is processed to obtain a processed image data packet, wherein the processing of the RGB image includes compressing the RGB image installation image quality and then encoding it according to the corresponding sending format, or encoding the original size according to the corresponding sending format;

[0018] Step 106: Establish a WebSocket-based communication module, connect to the ROS-Bridge server in the Linux platform ROS system, and publish a ROS topic data packet of type Image or a ROS topic data packet of type CompressedImage of Image, wherein the ROS topic data packet of type Image is an image data packet obtained based on an uncompressed image; and the ROS topic data packet of type CompressedImage is an image data packet obtained based on a compressed image;

[0019] Step 107: according to the set frame rate, set the timer, and publish the ROS topic data packet obtained in the previous step regularly through the communication module;

[0020] Step 108: The Linux platform ROS system uses the ROS-Bridge module to receive the ROS topic data packet;

[0021] Step 109: Present the RGB image data in the Rviz software by subscribing to a ROS topic data packet of type Image or a ROS topic data packet of type CompressedImage;

[0022] The depth image module process steps are as follows:

[0023] Step 201: Use the Camera component in the Unity3D three-dimensional game engine on the Windows platform to obtain the original visual field;

[0024] Step 202: Perform a linear projection operation on the original visual domain to obtain a depth visual domain, wherein when performing a linear projection operation, the depth value z under perspective projection is vjew It is expressed as the following formula:

[0025]

[0026] Where Far and Near are the distances between the far and near clipping planes, respectively, and are also the values ​​of the farthest depth of field and the nearest depth of field; d is the depth value of each pixel after spatial orthogonal projection for a single depth image, d = 0.5 × z ndc +0.5, z ndc It is the value of the Z-axis component of the vertex coordinate in NDC;

[0027] Step 203: modify the resolution, horizontal field of view angle, vertical field of view angle, depth of field, and refresh rate of the Camera component based on the depth visual field, where:

[0028] Modify the resolution of the Camera component according to the resolution in the actual camera parameters. The resolution is expressed as image width W × height H;

[0029] The vertical field of view angle in the actual camera parameters requires modification of the vertical field of view angle HFOV;

[0030] The horizontal field of view angle WFOV is calculated according to the relationship between the resolution W×H, the vertical field of view angle HFOV, the horizontal field of view angle WFOV and the focal length f of the camera, and the horizontal field of view angle WFOV of the Camera component is modified according to the calculated horizontal field of view angle WFOV;

[0031] Modify the depth of field according to the nearest depth of field and the farthest depth of field in the actual camera parameters;

[0032] Modify the refresh rate according to the corresponding requirements of the refresh rate in the actual camera parameters;

[0033] Step 204: Create a new canvas component, intercept the canvas component according to the resolution determined in step 102, obtain a canvas component consistent with the resolution, assign the obtained depth visible field pixels to the canvas component by row and column, and return the corresponding canvas component, which is the generated depth image;

[0034] Step 205: adding corresponding noise to the generated depth image;

[0035] Step 206: Perform spatial conversion calculation on the depth image obtained in the previous step to obtain point cloud data corresponding to the depth image;

[0036] Step 207: Process the point cloud data according to the ROS communication format to obtain a processed data packet;

[0037] Step 208: Establish a WebSocket-based communication module, connect to the ROS-Bridge server in the ROS system, and publish a ROS topic data packet of PointCloud2 type based on the data packet obtained in step 207;

[0038] Step 209: according to the frame rate requirement, a timer is set to periodically send ROS topic data packets, which are received by the Linux platform ROS system;

[0039] Step 210: Using the ROS-Bridge module in the Linux platform ROS system to receive the ROS topic data packet;

[0040] Step 211: Present the depth point cloud data in the Rviz software by subscribing to the ROS topic data packet of type PointCloud2.

[0041] Preferably, in step 102 and step 203, the relationship between the resolution W×H, the vertical field of view angle HFOV, the horizontal field of view angle WFOV and the camera focal length f conforms to the following formula:

[0042]

[0043]

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] 1) By adding optional image noise, the noise and other interference data that would appear in a real camera are simulated, thus improving the realism of the simulated camera.

[0046] 2) By processing and encoding RGB image data, ROS sensor data in Image format is generated, which contains complete unprocessed RGB image data, and the RGB raw data simulation of the RGB camera in Unity3D is realized.

[0047] 3) By compressing and encoding RGB image data, ROS sensor data in CompressedImage format was generated. The image compression rate can be dynamically modified and set through C# scripts, and the RGB compressed data simulation of the RGB camera in Unity3D was realized.

[0048] 4) Through the spatial transformation calculation of the depth image, the transformation from camera depth image to camera point cloud data is realized in Unity3D.

[0049] 5) By processing and encoding the point cloud data, ROS sensor data in PointCloud2 format is generated, which contains all the point cloud information within the camera's perspective, and realizes the point cloud data simulation of the depth camera in Unity3D.

[0050] 6) Through the C# communication code based on webSocket, the interconnection between Unity3D under Windows and the ROS-Bridge server in Linux was realized, and the data interaction problem between the Unity3D three-dimensional simulation environment in the non-ROS environment and the ROS environment was solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 For the depth camera process;

[0052] Figure 2 For the RGB camera process. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0054] The present invention provides a simulation method for a visual sensor in Unity3D of a ROS system. By means of Unity3D simulation, a visual sensor in a virtual environment is realized and communicates with an actual ROS system, thereby providing a key technical method for a robot simulation test environment.

[0055] The simulation method of the visual sensor under Unity3D of the ROS system of the present invention is mainly composed of two parts: the Unity3D three-dimensional game engine of the Windows platform (hereinafter referred to as "Unity") and the ROS system of the Linux platform (hereinafter referred to as "ROS system"), which can be divided into two modules according to the function: an RGB image module and a depth image module.

[0056] The RGB image module process steps are as follows:

[0057] Step 1: Use the Camera component in Unity to obtain the original visible area.

[0058] Step 2: Modify the parameters of the Camera component based on the original visual field, specifically modifying the resolution, horizontal and vertical field of view, depth of field, refresh rate, and image quality.

[0059] For resolution, it is mainly necessary to meet the resolution requirements in the actual camera parameters. According to the resolution in the actual camera parameters, enter the corresponding resolution size (image width W×height H) in the specified input box to complete the modification of the simulated RGB camera resolution. The modification principle is that each time the resolution is modified, a new RenderTexture that meets the resolution size (W×H) is created, and then the original Camera's targetTexture is specified as this RenderTexture to complete the overwriting.

[0060] For the vertical field of view HFOV, it is mainly necessary to meet the vertical field of view requirements in the actual camera parameters. According to the vertical field of view in the actual camera parameters, enter the corresponding vertical field of view in the specified input box to complete the modification of the vertical field of view of the simulated RGB camera. The modification principle is to modify the fieldOfView in the original Camera component and return it to the simulated RGB camera to complete the modification of the vertical field of view.

[0061] For the horizontal field of view, it is mainly necessary to satisfy the camera pinhole imaging principle. When the camera resolution and vertical field of view are determined, the horizontal field of view WFOV is calculated according to the relationship between the camera resolution (W×H), vertical field of view HFOV, horizontal field of view WFOV, and camera focal length f. The calculation formula is as follows:

[0062]

[0063]

[0064] The depth of field is divided into two parts: the nearest depth of field and the farthest depth of field. It mainly needs to meet the corresponding depth of field requirements in the actual camera parameters. According to the depth of field in the actual camera parameters, enter the corresponding nearest depth of field and farthest depth of field in the specified input box to complete the modification of the simulated RGB camera depth of field parameters. The modification principle is to modify the nearClipPlane and farClipPlane in the original Camera component and return the corresponding parameters to the simulated RGB camera.

[0065] For the refresh rate, it is mainly necessary to meet the corresponding refresh rate requirements in the actual camera parameters. According to the refresh rate in the actual camera parameters, enter the corresponding refresh rate in the specified input box to complete the modification of the refresh rate of the simulated RGB camera. The modification principle is to set the image frame interval to the corresponding refresh rate requirement. According to the frame interval, the corresponding image is obtained to complete the image acquisition requirement of the specified refresh rate.

[0066] For image quality, it is mainly necessary to meet the actual requirements of the enterprise algorithm and dynamically adjust the image quality according to the bandwidth. When the enterprise requires high quality and the bandwidth meets the requirements, the image quality is increased, and when the bandwidth is low, the image quality is reduced. The modification method is to enter the corresponding compression ratio in the specified input box. The modification principle is to compress the camera image according to the specified compression ratio and then send it to the corresponding enterprise.

[0067] Step 3: Create a canvas component with a freely adjustable aspect ratio to modify the original visual domain and obtain a color image. The modification principle is to create a large enough canvas component, intercept it according to the specified camera resolution parameter requirements, and obtain a canvas component with the same resolution as the RGB camera. Assign the original visual domain pixels obtained by the RGB camera to the canvas component by row and column, and return the corresponding canvas component, which is the generated RGB image.

[0068] Step 4: In order to restore the sense of reality, a variety of optional noises including salt and pepper noise and Gaussian noise are added to the RGB image.

[0069] Step 5: According to the ROS communication format, the RGB image with added noise is processed (compressed / uncompressed, encoded according to the corresponding sending format) to obtain a processed image data packet.

[0070] Step 6: Establish a WebSocket-based communication module, connect to the ROS-Bridge server in the ROS system, and publish ROS topic data packets of type Image and CompressedImage. The ROS topic of type Image is an image data packet obtained based on an uncompressed image. The ROS topic of type CompressedImage is an image data packet obtained based on a compressed image.

[0071] Step 7: Set the timer according to the set frame rate and publish the ROS topic data packet regularly through the communication module.

[0072] Step 8: Use the ROS-Bridge module in the ROS system to receive ROS topic data packets.

[0073] Step 9: Subscribe to the ROS topic (Image and CompressedImage types) of the RGB image module to present the RGB image data in the Rviz software. The two types of image data can be received at the same time, or one can be selected. Image is the original image without pixel distortion; CompressedImage is a compressed image compressed into jpg format, which has a certain degree of distortion. The degree of distortion is related to the compression selected in the Unity3D script.

[0074] The depth image module process steps are as follows:

[0075] Step 1: Use the Camera component in Unity to obtain the original visible area.

[0076] Step 2: According to the rendering principle in Shader, perform linear projection on the original visual domain to obtain the depth visual domain. The specific calculation method is as follows:

[0077] For a single depth image, after orthogonal projection, the depth value of each pixel corresponds to the value of the Z-axis component of the vertex coordinates in NDC. Because the NDC space value in Unity is [-1,1], the depth value d (that is, the pixel value in the depth texture) needs to be converted by the following formula:

[0078] d = 0.5 × z ndc +0.5

[0079] In the formula, z ndc Corresponds to the value of the Z component of the vertex coordinates in the NDC coordinates. At this time, the depth value of a single pixel vertex is obtained.

[0080] In Unity, perspective projection is nonlinear. Depth calculation for perspective projection will show that objects near are larger than objects far away. If you need to obtain a linear depth value, you need to further calculate the depth value in the perspective projection to get the depth value z under the perspective projection. view .

[0081] When using the clipping matrix P of the perspective projection clip After transforming the next pixel vertex in the view space, the z value of the vertex in the clipping space is clip and w clip They are:

[0082]

[0083] w clip =-z view

[0084] Among them, Far and Near are the distances of the far and near clipping planes, and are also the values ​​of the farthest depth of field and the nearest depth of field. Then, the upper component of NDC can be obtained by homogeneous division.

[0085]

[0086] For the depth texture depth value, the known relationship with NDC is:

[0087] d = 0.5 × z ndc +0.5

[0088] So we can conclude that:

[0089]

[0090] In Unity, the value of the Z component corresponding to the positive direction of the Camera component is negative. In order to make the depth value positive, the above result needs to be negated, and then:

[0091]

[0092] At this time, the value range is the frustum depth range [Near, Far]. The value range needs to be normalized later, even if the range is [0,1], 0 means that the point is at the same position as the camera, and 1 means that the point is on the far clipping plane Far of the frustum. Therefore, the above results need to be further settled:

[0093]

[0094] After the above steps, the depth visual field can be obtained.

[0095] Step 3: Modify the Camera parameters based on the depth of field, including resolution, horizontal and vertical field of view, depth of field, and refresh rate.

[0096] For resolution, it is mainly necessary to meet the resolution requirements in the actual camera parameters. According to the resolution in the actual camera parameters, enter the corresponding resolution size (image width W×height H) in the specified input box to complete the modification of the simulated depth camera resolution. The modification principle is that each time the resolution is modified, a new RenderTexture that meets the resolution size (W×H) is created, and then the original Camera's targetTexture is specified as this RenderTexture to complete the overwriting.

[0097] For the vertical field of view angle, it is mainly necessary to meet the vertical field of view angle requirements in the actual camera parameters. According to the vertical field of view angle in the actual camera parameters, enter the corresponding vertical field of view angle in the specified input box to complete the modification of the vertical field of view angle of the simulated depth camera. The modification principle is to modify the fieldOfView in the original Camera component and return it to the simulated RGB camera to complete the modification of the vertical field of view angle.

[0098] For the horizontal field of view, it is mainly necessary to satisfy the camera pinhole imaging principle. When the camera resolution and vertical field of view are determined, the horizontal field of view WFOV is calculated according to the relationship between the camera resolution W×H, the vertical field of view HFOV, the horizontal field of view WFOV, and the camera focal length f. The calculation formula is as follows:

[0099]

[0100]

[0101] The depth of field is divided into two parts: the nearest depth of field and the farthest depth of field. It mainly needs to meet the corresponding depth of field requirements in the actual camera parameters. According to the depth of field in the actual camera parameters, enter the corresponding nearest depth of field and farthest depth of field in the specified input box to complete the modification of the depth of field parameters of the simulated depth camera. The modification principle is to modify the nearClipPlane and farClipPlane in the original Camera component and return the corresponding parameters to the simulated depth camera.

[0102] For the refresh rate, it is mainly necessary to meet the corresponding refresh rate requirements in the actual camera parameters. According to the refresh rate in the actual camera parameters, enter the corresponding refresh rate in the specified input box to complete the modification of the refresh rate of the simulated depth camera. The modification principle is to set the image frame interval to the corresponding refresh rate requirement. According to the frame interval, the corresponding image is obtained to complete the image acquisition requirement of the specified refresh rate.

[0103] Step 4: Create a canvas component with a freely adjustable aspect ratio to modify the depth visual domain and obtain the depth image. The modification principle is to create a large enough canvas component, intercept it according to the specified camera resolution parameter requirements, and obtain a canvas component with the same resolution as the depth camera. Assign the depth visual domain pixels obtained by the depth camera to the canvas component by row and column, and return the corresponding canvas component, which is the generated depth image.

[0104] Step 5: In order to restore the sense of reality, a variety of optional noises including salt and pepper noise and Gaussian noise are added to the depth image.

[0105] Step 6: Perform spatial conversion calculation on the depth image to obtain point cloud data corresponding to the depth image. This embodiment provides two methods:

[0106] Method 1: Use the Unity3D function Vector3 ScreenToWorldPoint(Vector3 position) to extract each point in the camera depth image, calculate the world coordinates corresponding to the point through this function, and then pack the data of all points, that is, the point cloud data, into PointCloud2.

[0107] Method 2: Use the above camera parameters to write CameraInfo format data and depth image data in Unity3D (this data is also divided into Image and CompressedImage, including the depth image of the camera), and then send it to the ROS system through the communication module described in step 8. In the ROS system, after receiving the CameraInfo camera parameters and depth image data, the XYZ point cloud coordinates corresponding to each point on the depth image are calculated, all point cloud data are packaged, and the ROS topic of PointCloud2 is generated and published. Then proceed to step 9.

[0108] Referring to the pinhole imaging principle, the calculation formula and method of method 2 are as follows:

[0109] Through the CameraInfo camera parameters, obtain and generate the pinhole camera model PinholeCameraModel to obtain the horizontal and vertical coordinates of the image center of the camera intrinsic parameter matrix (center x ,center y ), get the f / dx(center x ) and f / dy(center y ). For each point on the depth image, there are coordinates (u,,epth), where u represents the height of the point in the image, v represents the width of the point in the image, and depth represents the depth of the point in the image (in the range of 0-65535, float format data). From this, the XYZ point cloud coordinates corresponding to the point can be calculated, and the coordinates need to be rotated:

[0110]

[0111] X=-(u-center x )×depth×center x

[0112] Y=depth

[0113] Z=-(v-center y )×depth×center y

[0114] Step 7: According to the ROS communication format, process the point cloud (encode it according to the corresponding sending format) to obtain the processed data packet.

[0115] Step 8: Establish a WebSocket-based communication module, connect to the ROS-Bridge server in the ROS system, and publish a ROS topic data packet of type PointCloud2.

[0116] Step 9: According to the frame rate requirements, set the timer to send ROS topic data packets regularly, and the ROS system receives them.

[0117] Step 10: Use the ROS-Bridge module in the ROS system to receive ROS topic data packets.

[0118] Step 11: Present the depth point cloud data in the Rviz software by subscribing to the point cloud topic (PointCloud2 type) of the depth image module.

Claims

1. A simulation method for a visual sensor in Unity3D suitable for a ROS system, characterized in that: Used to obtain RGB images or depth images, where: Obtaining an RGB image by the simulation method comprises the following steps: Step 101: Use the Camera component in the ∪nity3D three-dimensional game engine on the Windows platform to obtain the original visual field; Step 102: Modify the resolution, horizontal field of view angle, vertical field of view angle, depth of field, refresh rate, and image quality of the Camera component based on the original visual domain, where: Modify the resolution of the Camera component according to the resolution in the actual camera parameters. The resolution is expressed as image width W × height H; Modify the vertical field of view HFOV according to the vertical field of view requirements in the actual camera parameters; The horizontal field of view angle WFOV is calculated according to the relationship between the resolution W×H, the vertical field of view angle HFOV, the horizontal field of view angle WFOV and the focal length f of the camera, and the horizontal field of view angle WFOV of the Camera component is modified according to the calculated horizontal field of view angle WFOV; Modify the depth of field according to the nearest depth of field and the farthest depth of field in the actual camera parameters; Modify the refresh rate according to the corresponding requirements of the refresh rate in the actual camera parameters; Dynamically set the corresponding image compression ratio according to the bandwidth of the target user to adjust the image quality; Step 103: Create a new canvas component, intercept the canvas component according to the resolution determined in step 102, obtain a canvas component consistent with the resolution, assign the obtained original visible field pixels to the canvas component by row and column, and return the corresponding canvas component, which is the generated RGB image; Step 104: adding corresponding noise to the generated RGB image; Step 105: According to the Linux platform ROS system communication format, the RGB image obtained in the previous step is processed to obtain a processed image data packet, wherein the processing of the RGB image includes compressing the RGB image according to the image quality and then encoding it according to the corresponding transmission format, or encoding the original size according to the corresponding transmission format; Step 106: Establish a WebSocket-based communication module, connect to the ROS-Bridge server in the Linux platform ROS system, and publish a ROS topic data packet of type Image or a ROS topic data packet of type CompressedImage, wherein the ROS topic data packet of type Image is an image data packet obtained based on an uncompressed image; the ROS topic data packet of type CompressedImage is an image data packet obtained based on a compressed image; Step 107: according to the set frame rate, set the timer, and publish the ROS topic data packet obtained in the previous step regularly through the communication module; Step 108: The Linux platform ROS system uses the ROS-Bridge module to receive the ROS topic data packet; Step 109: presenting the RGB image data in the Rviz software by subscribing to a ROS topic data packet of type Image or a ROS topic data packet of type CompressedImage. The depth image module process steps are as follows: Step 201: Use the Camera component in the Unity3D three-dimensional game engine on the Windows platform to obtain the original visual field; Step 202: Perform a linear projection operation on the original visual domain to obtain a depth visual domain, wherein when performing a linear projection operation, the depth value z under perspective projection is view It is expressed as the following formula: Where Far and Near are the distances between the far and near clipping planes, respectively, and are also the values ​​of the farthest depth of field and the nearest depth of field; d is the depth value of each pixel after spatial orthogonal projection for a single depth image, d = 0.5 × z ndc +0.5, z ndc It is the value of the Z-axis component of the vertex coordinate in NDC; Step 203: modify the resolution, horizontal field of view angle, vertical field of view angle, depth of field, and refresh rate of the Camera component based on the depth visual field, where: Modify the resolution of the Camera component according to the resolution in the actual camera parameters. The resolution is expressed as image width W × height H; Modify the vertical field of view HFOV according to the vertical field of view requirements in the actual camera parameters; The horizontal field of view angle WFOV is calculated according to the relationship between the resolution W×H, the vertical field of view angle HFOV, the horizontal field of view angle WFOV and the focal length f of the camera, and the horizontal field of view angle WFOV of the Camera component is modified according to the calculated horizontal field of view angle WFOV; Modify the depth of field according to the nearest depth of field and the farthest depth of field in the actual camera parameters; Modify the refresh rate according to the corresponding requirements of the refresh rate in the actual camera parameters; Step 204: Create a new canvas component, intercept the canvas component according to the resolution determined in step 102, obtain a canvas component consistent with the resolution, assign the obtained depth visible field pixels to the canvas component by row and column, and return the corresponding canvas component, which is the generated depth image; Step 205: adding corresponding noise to the generated depth image; Step 206: Perform spatial conversion calculation on the depth image obtained in the previous step to obtain point cloud data corresponding to the depth image; Step 207: Process the point cloud data according to the ROS communication format to obtain a processed data packet; Step 208: Establish a WebSocket-based communication module, connect to the ROS-Bridge server in the ROS system, and publish a ROS topic data packet of PointCloud2 type based on the data packet obtained in step 207; Step 209: according to the frame rate requirement, a timer is set to periodically send ROS topic data packets, which are received by the Linux platform ROS system; Step 210: Using the ROS-Bridge module in the Linux platform ROS system to receive the ROS topic data packet; Step 211: Present the depth point cloud data in the Rviz software by subscribing to the ROS topic data packet of type PointCloud2.

2. The simulation method of a visual sensor in Unity3D applicable to a ROS system as claimed in claim 1, characterized in that: In step 102 and step 203, the relationship between the resolution W×H, the vertical field of view HFOV, the horizontal field of view WFOV and the focal length f of the camera conforms to the following formula:

Images