Method for simulating and generating sea ice area on sea surface

Abstract

The invention relates to a method for simulating and generating a sea ice area on a sea surface to obtain sea ice distribution data meeting actual requirements, and relates to the technical field of image processing. The method comprises the following steps: a first part: perspective projection of a two-dimensional sea surface image; the second part is projection of the three-dimensional model to a two-dimensional plane; and in the third part, the projected images of the first part and the second part are combined, and corresponding simulation information is added. According to the method for simulating and generating the sea ice area on the sea surface, an effective data set can be obtained, and subsequent deep learning or machine learning training can be conveniently carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for simulating and generating sea ice areas on a sea surface. Background Art

[0002] Based on the fact that some projects need a data set of the distribution of sea ice in cold areas on the sea surface, if there is a data set, it can be well simulated that the area observed by the ship's camera when the ship enters the sea ice area, and better route selection training can be carried out according to the simulated data set, and obstacles can be better avoided, risks can be better avoided, and project requirements can be better completed and project goals can be achieved. There is no data available on the Internet, and the cost of actually shooting at sea is very expensive, so how to get such a usable data set has become a big problem. Summary of the invention

[0003] The present invention aims to solve the technical problems in the prior art, obtain sea ice distribution data that meets actual requirements, and provide a method for simulating and generating sea ice areas on the sea surface.

[0004] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0005] A method for simulating and generating sea ice areas on a sea surface comprises the following steps:

[0006] Part I: Perspective projection of two-dimensional sea surface images;

[0007] Part II: Projection of a 3D model onto a 2D plane;

[0008] Part 3: Merge the projected images of the first and second parts and add corresponding simulation information.

[0009] In the above technical solution, the first part includes step one and step two:

[0010] Step 1: Simulate the shooting state of the camera, set the internal and external parameters of the camera model, simulate the ideal camera model state, and set the adjustable parameters: focal length, tilt and rotation angle;

[0011] Step 2: Perform perspective transformation on the photograph of the sea surface; adjust the tilt and rotation angles, generate the tilt matrix R_x around the x-axis and the rotation matrix R_z around the z-axis according to the adjusted angles, and output a combined rotation matrix R after multiplying the two matrices. The combined rotation matrix, camera parameter matrix and distortion parameter matrix are applied as parameters to the cv2.initUndistortRectifyMap function to output two mapping matrices mapx and mapy. These two mapping matrices are passed as parameters to the cv2.remap function to output the result of the perspective transformation, simulating the effect that objects appear larger when near and smaller when far away as seen by the human eye.

[0012] In the above technical solution, among the adjustable parameters in step 1, the tilt angle is the angle of transformation along the x-axis, and the rotation angle is the angle of transformation along the z-axis.

[0013] In the above technical solution, in step 1, the internal parameters of the camera model are:

[0014] [[focal_length,0,cx],

[0015] [0,focal_length,cy],

[0016] [0,0,1]]

[0017] Among them, focal_length is the focal length of the camera model, cx and cy are the center positions of the sea surface photo respectively;

[0018] The tilt matrix around the x-axis is:

[0019] R_x = np.array([

[0020] [1,0,0],

[0021] [0,np.cos(tilt_rad),-np.sin(tilt_rad)],

[0022] [0,np.sin(tilt_rad),np.cos(tilt_rad)]])

[0023] The rotation matrix around the z-axis is:

[0024] R_z = np.array([

[0025] [np.cos(rotation_rad),-np.sin(rotation_rad),0],

[0026] [np.sin(rotation_rad),np.cos(rotation_rad),0],

[0027] [0,0,1]]);

[0028] The combined rotation matrix R satisfies: R = R_x@R_z.

[0029] In the above technical solution, the process of the perspective transformation function in step 2 includes:

[0030] Initialize the mapping coordinates and apply the cv2.initUndistortRectifyMap function to return two mapping matrices mapx and mapy;

[0031] Remap the image and apply the cv2.remap function to remap the input distorted image to the corrected image according to mapx and mapy.

[0032] In the above technical solution, the second part includes step three and step four:

[0033] Step 3: Find a 3D model to render and simulate icebergs and floating ice. Use a rockery model to render and simulate icebergs, and use lawn to simulate floating ice.

[0034] Step 4: Project the rendered 3D model onto a 2D plane, set the camera's intrinsic and extrinsic parameters, and then set the adjustable parameters: the camera's position and posture. The posture includes the offset angles around the x and y axes, which are also expressed as multiplication in the code. According to the posture matrix, simulate the camera's shooting angle and project the 3D model onto a 2D plane.

[0035] In the above technical solution, the intrinsic parameter of the camera in step 4 is the IntrinsicsCamera function in the pyrender library, which is set to IntrinsicsCamera = (fx = 1600, fy = 1600, cx = w / 2, cy = h / 2), where w and h are the length and width of the image on the two-dimensional plane;

[0036] The tilt_angle and rotation_angle of the 3D model projected onto the 2D plane are the angles of tilt around the x-axis and rotation around the y-axis, which are converted into radians to calculate the tilt and rotation matrices;

[0037] The rotation matrix around the x-axis is:

[0038] tilt_matrix = np.array([

[0039] [1,0,0],

[0040] [0,np.cos(tilt_rad),-np.sin(tilt_rad)],

[0041] [0,np.sin(tilt_rad),np.cos(tilt_rad)]])

[0042] The rotation matrix around the y-axis is:

[0043] rotation_matrix = np.array([

[0044] [np.cos(rotation_rad),0,np.sin(rotation_rad)],

[0045] [0,1,0],

[0046] [-np.sin(rotation_rad),0,np.cos(rotation_rad)]])

[0047] Combine the rotation matrix expansion and the camera position matrix to form the camera pose matrix;

[0048] After the camera's pose matrix is ​​set, use the Mesh, Scene, and OffscreenRenderer functions of the pyrender library to project the three-dimensional model onto a two-dimensional plane and save the output.

[0049] In the above technical solution, the third part includes: step five;

[0050] Step 5: After appropriate shearing, the perspective transformed image of the two-dimensional image is merged with the image of the three-dimensional model projected onto the two-dimensional plane to form an image simulating the sea ice area, and then the corresponding simulation information is added.

[0051] In the above technical solution, the corresponding simulation information includes: location, time and distance information.

[0052] The present invention has the following beneficial effects:

[0053] By using the method of simulating and generating sea ice areas on the sea surface of the present invention, an effective data set can be obtained, which is convenient for subsequent deep learning or machine learning training. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0055] Figure 1 This is the original sea surface photo taken.

[0056] Figure 2 This is a picture that is obtained by perspective transforming the two-dimensional sea surface picture.

[0057] Figure 3 for Figure 2 The image is obtained by cropping the image.

[0058] Figure 4 It is a two-dimensional image of the rockery after rendering and projection onto a two-dimensional plane.

[0059] Figure 5 This is a two-dimensional image of the lawn that is projected onto a two-dimensional plane after rendering.

[0060] Figure 6 for Figure 4 and Figure 3 The image after merging.

[0061] Figure 7 for Figure 6 and Figure 5 The image after merging.

[0062] Figure 8 A schematic flow chart of the steps of the method for simulating and generating sea ice areas on the sea surface according to the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below with reference to the accompanying drawings.

[0064] like Figure 1-8 As shown, the method of simulating and generating sea ice area on the sea surface of the present invention is mainly divided into three parts:

[0065] The first part is the perspective projection of the two-dimensional sea surface image; the second part is the projection of the three-dimensional model onto the two-dimensional plane; the third part is to merge the projected images of the first and second parts and simulate the position, time and distance information. Figure 1 This is the original sea surface photo taken.

[0066] Specifically:

[0067] The first part, perspective projection of a two-dimensional sea surface image, includes: step 1 and step 2, wherein:

[0068] Step 1: Simulate the shooting state of the camera, set the intrinsic and extrinsic parameters of the camera model, simulate the ideal camera model state, and set the adjustable parameters: focal length, tilt and rotation angle. The tilt angle is the angle of transformation along the x-axis, and the rotation angle is the angle of transformation along the z-axis. A certain angle requires the superposition of tilt and rotation, which is expressed as multiplication in the code.

[0069] Specifically, simulate the shooting state of the camera and set the parameters of the camera model. First, the parameters of the camera are:

[0070] [[focal_length,0,cx],

[0071] [0,focal_length,cy],

[0072] [0,0,1]]

[0073] Among them, focal_length is the focal length of the camera model, which is an adjustable parameter, and cx, cy are set to the center position of the sea surface photo. Then, assuming that the camera has no distortion, because it is an ideal new state, the distortion is set to zero matrix.

[0074] Step 2: Perform perspective transformation on the photograph of the sea surface; adjust the tilt and rotation angles, generate the tilt matrix R_x around the x-axis and the rotation matrix R_z around the z-axis according to the adjusted angles, and output a combined rotation matrix R after multiplying the two matrices. The combined rotation matrix, camera parameter matrix and distortion parameter matrix are applied as parameters to the cv2.initUndistortRectifyMap function to output two mapping matrices mapx and mapy. These two mapping matrices are passed as parameters to the cv2.remap function to output the result of the perspective transformation, simulating the effect that objects appear larger when near and smaller when far away as seen by the human eye.

[0075] Specifically, set the angles tilt_angle and rotation_angle to the angles of tilt around the x-axis and rotation around the z-axis, respectively, convert them to radians, and calculate the tilt and rotation matrices.

[0076] The tilt matrix around the x-axis is:

[0077] R_x = np.array([

[0078] [1,0,0],

[0079] [0,np.cos(tilt_rad),-np.sin(tilt_rad)],

[0080] [0,np.sin(tilt_rad),np.cos(tilt_rad)]])

[0081] The rotation matrix around the z-axis is:

[0082] R_z = np.array([

[0083] [np.cos(rotation_rad),-np.sin(rotation_rad),0],

[0084] [np.sin(rotation_rad),np.cos(rotation_rad),0],

[0085] [0,0,1]])

[0086] The combined rotation matrix R satisfies: R = R_x@R_z.

[0087] Then, a perspective transformation is performed, and the image after the perspective transformation is shown in Figure 2 .

[0088] The first step is to initialize the mapping coordinates. Applying the cv2.initUndistortRectifyMap function returns two mapping matrices, mapx and mapy, which are used to map the distorted image to the corrected image. mapx, mapy = cv2.initUndistortRectifyMap(camera_matrix,dist_coeffs,R,camera_matrix,image_size,5), where camera_matrix is ​​the camera parameter matrix, dist_coeffs is the distortion matrix, R is the combined rotation matrix R generated in step 2, camera_matrix (the second one) is the camera intrinsic parameter matrix of the output image, which is usually the same as the input, image_size is the size of the output image, and 5 means the type of the output image, which is usually CV_32FC1.

[0089] The second step is to remap the image. Use the cv2.remap function to remap the input distorted image to the corrected image dst according to mapx and mapy. dst = cv2.remap (image, mapx, mapy, cv2.INTER_LINEAR), image is the input distorted image, mapx and mapy are the mapping matrices generated in the previous step, cv2.INTER_LINEAR: interpolation method, used for pixel value interpolation when remapping the image. INTER_LINEAR is a bilinear interpolation method.

[0090] The second part, projection of the three-dimensional model onto a two-dimensional plane, includes: step three and step four, wherein:

[0091] Step 3: Find a three-dimensional model to render and simulate icebergs and floating ice. Use a rockery model to render and simulate icebergs, and use lawns to simulate floating ice.

[0092] Step 4: Project the rendered 3D model onto a 2D plane; Same as step 1, set the camera's intrinsic and extrinsic parameters, and then set the adjustable parameters: the camera's position and attitude. The attitude includes the offset angle around the x and y axes, which is also expressed as multiplication in the code. According to the attitude matrix, simulate the camera's shooting angle and project the 3D model onto a 2D plane.

[0093] Specifically, set the intrinsic parameters of the camera. The intrinsic parameters of the camera are the IntrinsicsCamera function in the pyrender library, which is set to IntrinsicsCamera = (fx = 1600, fy = 1600, cx = w / 2, cy = h / 2), where w and h are the length and width of the image in the two-dimensional plane.

[0094] Set the angles tilt_angle1 and rotation_angle1 to the angles of tilt around the x-axis and rotation around the y-axis respectively, convert them to radians, and calculate the tilt and rotation matrices.

[0095] The rotation matrix around the x-axis is:

[0096] tilt_matrix = np.array([

[0097] [1,0,0],

[0098] [0,np.cos(tilt_rad),-np.sin(tilt_rad)],

[0099] [0,np.sin(tilt_rad),np.cos(tilt_rad)]])

[0100] The rotation matrix around the y-axis is:

[0101] rotation_matrix = np.array([

[0102] [np.cos(rotation_rad),0,np.sin(rotation_rad)],

[0103] [0,1,0],

[0104] [-np.sin(rotation_rad),0,np.cos(rotation_rad)]])

[0105] Combine the rotation matrix R = rotation_matrix@tilt_matrix, expand and add the camera position matrix T to form the camera pose matrix RT.

[0106] Use the Mesh, Scene, and OffscreenRenderer functions of the pyrender library to project the 3D model onto a 2D plane and save the output. See the projected image. Figure 4 , Figure 5 . First, scene_mesh =

[0107] Mesh.from_trimesh(mesh), mesh: represents a 3D geometric model.

[0108] Mesh.from_trimesh(mesh): Converts the Trimesh object to the mesh object scene_mesh in the rendering library. Then creates a scene and sets the ambient light and background color: scene =

[0109] Scene(ambient_light=np.array([1.0,1.0,1.0,1.0]),bg_color=[1,0,0,1]),ambient_light: the ambient light color in the scene,bg_color: the background color of the scene. Add the mesh to the scene: scene.add(scene_mesh), Add the camera to the scene:

[0110] scene.add(camera_intrinsics,pose=camera_pose), camera_intrinsics: camera intrinsics object, pose=camera_pose: camera pose matrix. Finally, create an off-screen renderer and render the scene: renderer=OffscreenRenderer(w,h), create an off-screen renderer with a viewport size of width w and height h, color_image,_=renderer.render(scene), render the scene scene, and return the projected image.

[0111] The third part combines the projected images of the first and second parts and simulates the position, time and distance information, including: step five.

[0112] Step 5: After appropriate shearing, the perspective transformed image of the two-dimensional image is merged with the image projected onto the two-dimensional plane to form an image simulating the sea ice area on the sea surface, and then the required corresponding simulation information is added.

[0113] Specifically, the perspective transformed image of the two-dimensional image is appropriately sheared, see Figure 3 , and merged with the image projected from three dimensions to a two-dimensional plane. The merged image is shown in Figure 6 and Figure 7 .

[0114] First, ensure that the image size after the 3D model is projected onto the 2D plane is the same as the background image. This requires setting w and h in renderer = OffscreenRenderer (w, h) to be equal to the length and width of the background image.

[0115] Then, create a mask, find all the black pixels in the image projected from three dimensions to the two-dimensional plane, fill the black areas in the appropriately sheared two-dimensional image, and get a new image after merging the two images.

[0116] The merged sea ice image is simulated to generate the shooting location, time, and the distance between the shooting location and the sea ice location.

[0117] By using the method of simulating and generating sea ice areas on the sea surface of the present invention, an effective data set can be obtained, which is convenient for subsequent deep learning or machine learning training.

[0118] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A method for simulating and generating sea ice areas on the sea surface, characterized in that: The following steps are involved: Part I: Perspective projection of two-dimensional sea surface images; Part II: Projection of a 3D model onto a 2D plane; Part 3: Merge the projected images of the first and second parts and add corresponding simulation information.

2. The method for simulating and generating sea ice areas on the sea surface according to claim 1, characterized in that: The first part includes steps 1 and 2: Step 1: Simulate the shooting state of the camera, set the internal and external parameters of the camera model, simulate the ideal camera model state, and set the adjustable parameters: focal length, tilt and rotation angle; Step 2: Perform perspective transformation on the photograph of the sea surface; adjust the tilt and rotation angles, and generate the tilt matrix R_x around the x-axis and the rotation matrix R_z around the z-axis according to the adjusted angles. After multiplying the two matrices, a combined rotation matrix R is output. The combined rotation matrix, camera parameter matrix, and distortion parameter matrix are applied as parameters to the cv2.initUndistortRectifyMap function to output two mapping matrices mapx and mapy. These two mapping matrices are passed as parameters to the cv2.remap function to output the result of the perspective transformation, simulating the effect that objects appear larger when near and smaller when far away as seen by the human eye.

3. The method for simulating and generating sea ice areas on the sea surface according to claim 2, characterized in that: Among the adjustable parameters in step 1, the tilt angle is the angle of transformation along the x-axis, and the rotation angle is the angle of transformation along the z-axis.

4. The method for simulating and generating sea ice areas on the sea surface according to claim 2, characterized in that: In step 1, the intrinsic parameters of the camera model are: [[focal_length,0,cx], [0,focal_length,cy], [0,0,1]] Among them, focal_length is the focal length of the camera model, cx and cy are the center positions of the sea surface photo respectively; The tilt matrix around the x-axis is: R_x = np.array([ [1,0,0], [0,np.cos(tilt_rad),-np.sin(tilt_rad)], [0,np.sin(tilt_rad),np.cos(tilt_rad)]]) The rotation matrix around the z-axis is: R_z = np.array([ [np.cos(rotation_rad),-np.sin(rotation_rad),0], [np.sin(rotation_rad),np.cos(rotation_rad),0], [0,0,1]])； The combined rotation matrix R satisfies: R = R_x@R_z.

5. The method for simulating and generating sea ice areas on the sea surface according to claim 2, characterized in that: The process of the perspective transformation function in step 2 includes: Initialize the mapping coordinates and apply the cv2.initUndistortRectifyMap function to return two mapping matrices mapx and mapy; Remap the image and apply the cv2.remap function to remap the input distorted image to the corrected image according to mapx and mapy.

6. The method for simulating and generating sea ice areas on the sea surface according to claim 1, characterized in that: The second part includes steps three and four: Step 3: Find a 3D model to render and simulate icebergs and floating ice. Use a rockery model to render and simulate icebergs, and use lawn to simulate floating ice. Step 4: Project the rendered 3D model onto a 2D plane, set the camera's intrinsic and extrinsic parameters, and then set the adjustable parameters: the camera's position and posture. The posture includes the offset angles around the x and y axes, which are also expressed as multiplication in the code. According to the posture matrix, simulate the camera's shooting angle and project the 3D model onto a 2D plane.

7. The method for simulating and generating sea ice areas on the sea surface according to claim 6, characterized in that: In step 4, the intrinsic parameters of the camera are the IntrinsicsCamera function in the pyrender library, which is set to IntrinsicsCamera = (fx = 1600, fy = 1600, cx = w / 2, cy = h / 2), where w and h are the length and width of the image on the two-dimensional plane; The tilt_angle and rotation_angle of the 3D model projected onto the 2D plane are the angles of tilt around the x-axis and rotation around the y-axis, which are converted into radians to calculate the tilt and rotation matrices; The rotation matrix around the x-axis is: tilt_matrix = np.array([ [1,0,0], [0,np.cos(tilt_rad),-np.sin(tilt_rad)], [0,np.sin(tilt_rad),np.cos(tilt_rad)]]) The rotation matrix around the y-axis is: rotation_matrix = np.array([ [np.cos(rotation_rad),0,np.sin(rotation_rad)], [0,1,0], [-np.sin(rotation_rad),0,np.cos(rotation_rad)]]) Combine the rotation matrix expansion and the camera position matrix to form the camera pose matrix; After the camera's pose matrix is ​​set, use the Mesh, Scene, and OffscreenRenderer functions of the pyrender library to project the three-dimensional model onto a two-dimensional plane and save the output.

8. The method for simulating and generating sea ice areas on the sea surface according to claim 1, characterized in that: The third part includes: Step 5; Step 5: After appropriate shearing, the perspective transformed image of the two-dimensional image is merged with the image of the three-dimensional model projected onto the two-dimensional plane to form an image simulating the sea ice area, and then the corresponding simulation information is added.

9. The method for simulating and generating sea ice areas on the sea surface according to claim 8, characterized in that: The corresponding simulation information includes: location, time and distance information.

Images