A method of simulating the generation of sea ice regions on the sea surface

Abstract

The present application relates to a kind of methods for simulating the sea ice area of sea surface, to obtain such actual requirements of sea ice distribution data, it is related to the field of image processing.The method comprises the following steps: first part: the perspective projection of two-dimensional sea surface image;Second part: the projection of three-dimensional model to two-dimensional plane;Third part: the image of the projection of the first and second two parts is merged, and corresponding simulation information is added.Through the method for simulating the sea ice area of sea surface of the present application, effective data set can be obtained, and subsequent deep learning or machine learning training is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a method for simulating generation of sea ice area on sea surface. BACKGROUND

[0002] Based on the dataset of the distribution of cold areas of sea ice on sea surface in some projects, if the dataset is available, the area observed by the camera on the ship entering the sea ice area can be well simulated, the route selection training can be better, the obstacles can be better avoided, the risks can be better avoided, the project requirements can be better met, and the project goals can be achieved. There is no available data on the network, and the cost of actual shooting at sea is very high, so how to obtain such a usable dataset is a big problem. SUMMARY

[0003] The present application solves the technical problems in the prior art, obtains such sea ice distribution data that meets the actual requirements, and provides a method for simulating generation of sea ice area on sea surface.

[0004] In order to solve the above technical problems, the technical scheme of the present application is as follows:

[0005] A method for simulating generation of sea ice area on sea surface, comprising the following steps:

[0006] First part: perspective projection of two-dimensional sea surface image;

[0007] Second part: projection of three-dimensional model to two-dimensional plane;

[0008] Third part: merging the projected images of the first and second parts and adding corresponding simulation information.

[0009] In the above technical scheme, the first part includes steps one and two:

[0010] Step one: simulate the shooting state of the camera, set the internal and external parameters of the camera model, simulate the ideal camera model state, and set adjustable parameters: focal length, tilt and rotation angle;

[0011] Step two: perspective transformation is performed on the photographed sea surface photo; the tilt and rotation angles are adjusted, a tilt matrix R_x around the x axis and a rotation matrix R_z around the z axis are generated according to the adjusted angles, the two matrices are multiplied to output a combined rotation matrix R, and the combined rotation matrix, the camera parameter matrix and the distortion parameter matrix are applied to the cv2.initUndistortRectifyMap function as parameters to output two mapping matrices mapx and mapy, which are input into the cv2.remap function as parameters to output the perspective transformation result, and the effect of the human eye being larger near and smaller far is simulated.

[0012] In the above technical solution, in the adjustable parameters in step one, the tilt angle is the angle of transformation along the x-axis, and the rotation angle is the angle of transformation along the z-axis.

[0013] In the above technical solution, in step one, the intrinsic parameter of the camera model is:

[0014] [[focal_length, 0, cx],

[0015] [0, focal_length, cy],

[0016] [0, 0, 1]]

[0017] Wherein, focal_length is the focal length of the set camera model, and cx, cy are the center positions of the sea surface photo respectively;

[0018] The tilt matrix around the x-axis is:

[0019] R_x = np.array([

[0020] [1, 0, 0],

[0021] [0, np.cos(tilt_rad), -np.sin(tilt_rad)],

[0022] [0, np.sin(tilt_rad), np.cos(tilt_rad)]])

[0023] The rotation matrix around the z-axis is:

[0024] R_z = np.array([

[0025] [np.cos(rotation_rad), -np.sin(rotation_rad), 0],

[0026] [np.sin(rotation_rad), np.cos(rotation_rad), 0],

[0027] [0, 0, 1]]);

[0028] The combined rotation matrix R satisfies: R = R_x@R_z.

[0029] In the above technical solution, the process of the perspective transformation function in step two includes:

[0030] Initialize the mapping coordinates, and apply the cv2.initUndistortRectifyMap function to return two mapping matrices mapx and mapy;

[0031] Remap the image, apply the cv2.remap function to remap the input distorted image to the corrected image according to mapx and mapy.

[0032] In the above technical solution, the second part includes steps three and four:

[0033] Step three: find a three-dimensional model to render a simulated iceberg and ice floe, render a simulated iceberg with a rockery model, and simulate an ice floe with a lawn;

[0034] Step four: project the rendered three-dimensional model to a two-dimensional plane, set the intrinsic and extrinsic parameters of the camera, and then set the adjustable parameters: the position and pose of the camera, the pose including the offset angle around the x and y axes, which is also represented as multiplication in the code; according to the pose matrix, simulate the camera shooting angle, and project the three-dimensional model to a two-dimensional plane.

[0035] In the above technical solution, the intrinsic parameter of the camera in step four is the IntrinsicsCamera function in the pyrender library, which is set to IntrinsicsCamera=(fx=1600, fy=1600, cx=w / 2, cy=h / 2), w and h are the length and width of the two-dimensional plane image;

[0036] The tilt_angle and rotation_angle of the three-dimensional model projected to the two-dimensional plane are the angles of tilting around the x axis and rotating around the y axis, converted to radians, and the tilt and rotation matrices are calculated;

[0037] The rotation matrix around the x axis is:

[0038] tilt_matrix = np.array([

[0039] [1, 0, 0],

[0040] [0, np.cos(tilt_rad), -np.sin(tilt_rad)],

[0041] [0, np.sin(tilt_rad), np.cos(tilt_rad)]])

[0042] The rotation matrix around the y axis is:

[0043] rotation_matrix = np.array([

[0044] [np.cos(rotation_rad), 0, np.sin(rotation_rad)],

[0045] [0, 1, 0],

[0046] [-np.sin(rotation_rad), 0, np.cos(rotation_rad)]])

[0047] The combined rotation matrix is expanded plus the camera position matrix to form the pose matrix of the camera.

[0048] After the pose matrix of the camera is set, the Mesh, Scene, OffscreenRenderer functions of the pyrender library are applied to project the three-dimensional model to the two-dimensional plane, and the output is saved.

[0049] In the above technical solution, the third part comprises: step five;

[0050] Step five: the image after perspective transformation of the two-dimensional image is moderately cut and combined with the image of the three-dimensional model projected to the two-dimensional plane to form an image simulating the photographed sea ice area, and then corresponding simulation information is added.

[0051] In the above technical solution, the corresponding simulation information includes: position, time and distance information.

[0052] The present application has the following beneficial effects:

[0053] Through the method for simulating the sea ice area on the sea surface of the present application, an effective data set can be obtained, and subsequent deep learning or machine learning training is facilitated. BRIEF DESCRIPTION OF DRAWINGS

[0054] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0055] Figure 1 The original sea surface photo obtained by shooting.

[0056] Figure 2 The picture obtained by perspective transformation of the two-dimensional sea surface picture.

[0057] Figure 3 The picture obtained by cutting the picture of Figure 2

[0058] The two-dimensional picture of the rockery projected to the two-dimensional plane after rendering. Figure 4

[0059] The two-dimensional picture of the lawn projected to the two-dimensional plane after rendering. Figure 5

[0060] The picture obtained by cutting the picture of Figure 6 Figure 4 Figure 3 The picture obtained by combining the picture of​​

[0061] Figure 7 For Figure 6 With Figure 5 the picture formed after merging.

[0062] Figure 8 The flow chart of the method for simulating the sea ice area on the sea surface according to the present application. DETAILED DESCRIPTION

[0063] The present application will be described in detail below with reference to the accompanying drawings.

[0064] As Figures 1-8 shown, the method for simulating the sea ice area on the sea surface according to the present application mainly includes three parts:

[0065] The first part is the perspective projection of the two-dimensional sea surface image; the second part is the projection of the three-dimensional model to the two-dimensional plane; and the third part is to merge the projected images of the first and second parts and simulate the position, time and distance information. Figure 1 The original sea surface photo obtained by shooting.

[0066] Specifically:

[0067] The first part, the perspective projection of the two-dimensional sea surface image, includes step one and step two, in which:

[0068] Step one: simulate the shooting state of the camera, set the intrinsic and extrinsic parameters of the camera model, simulate the ideal camera model state, and set the adjustable parameters: focal length, tilt angle and rotation angle. The tilt angle is the angle of transformation along the x-axis, and the rotation angle is the angle of transformation along the z-axis. The superposition of a certain angle needs to be tilted and rotated, which is represented as multiplication in the code.

[0069] Specifically, the shooting state of the camera is simulated, and the parameters of the camera model are set. First, the parameters of the camera are:

[0070] [[focal_length, 0, cx],

[0071] [0, focal_length, cy],

[0072] [0, 0, 1]]

[0073] Wherein, focal_length is the focal length of the set camera model, which is an adjustable parameter, and cx, cy are set as the center position of the sea surface photo. Then it is assumed that the camera has no distortion, because it is an ideal new state, so the distortion is set to a zero matrix.

[0074] Step two: perspective transformation is performed on the photographed sea surface photo; the tilt and rotation angles are adjusted, and a tilt matrix R_x around the x-axis and a rotation matrix R_z around the z-axis are generated according to the adjusted angles, the two matrices are multiplied to output a combined rotation matrix R, the combined rotation matrix, the camera parameter matrix and the distortion parameter matrix are applied to the cv2.initUndistortRectifyMap function as parameters to output two mapping matrices mapx and mapy, and the two mapping matrices are transmitted into the cv2.remap function as parameters to output the result of perspective transformation, which simulates the effect of human eyes that the near is large and the far is small.

[0075] Specifically, the angles tilt_angle and rotation_angle are set as the angles of tilt around the x-axis and rotation around the z-axis, which are converted into radians, and the tilt and rotation matrices are calculated.

[0076] The tilt matrix around the x-axis is:

[0077] R_x = np.array([

[0078] [1, 0, 0],

[0079] [0, np.cos(tilt_rad), -np.sin(tilt_rad)],

[0080] [0, np.sin(tilt_rad), np.cos(tilt_rad)]])

[0081] The rotation matrix around the z-axis is:

[0082] R_z = np.array([

[0083] [np.cos(rotation_rad), -np.sin(rotation_rad), 0],

[0084] [np.sin(rotation_rad), np.cos(rotation_rad), 0],

[0085] [0, 0, 1]])

[0086] The combined rotation matrix R satisfies: R = R_x@R_z.

[0087] Then, perspective transformation is performed, and the photo after perspective transformation is shown in Figure 2 .

[0088] The first step is to initialize the mapping coordinates, and the cv2.initUndistortRectifyMap function will return two mapping matrices mapx and mapy, which are used to map the distorted image to the corrected image, mapx, mapy = cv2.initUndistortRectifyMap(camera_matrix, dist_coeffs, R, camera_matrix, image_size, 5), where camera_matrix is the camera parameter matrix, dist_coeffs is the distortion matrix, R is the combined rotation matrix R generated in step two, camera_matrix (second) is the camera intrinsic matrix of the output image, usually the same as the input, image_size is the size of the output image, and 5 means the type of the output image, usually CV_32FC1.

[0089] The second step is to remap the image, and the cv2.remap function is used to remap the input distorted image to the corrected image dst according to mapx and mapy, dst = cv2.remap(image, mapx, mapy, cv2.INTER_LINEAR), image is the input distorted image, mapx and mapy are the mapping matrices generated in the previous step, cv2.INTER_LINEAR: interpolation method, used for pixel value interpolation when remapping the image. INTER_LINEAR is a bilinear interpolation method.

[0090] The second part, the projection of three-dimensional model to two-dimensional plane, includes step three and step four, wherein:

[0091] Step three: find the three-dimensional model rendering simulation iceberg and ice, use the rock model to render the simulation iceberg, and use the lawn to simulate the ice.

[0092] Step four: project the rendered three-dimensional model to a two-dimensional plane; the same as step one, set the camera's intrinsic and extrinsic parameters, and then set the adjustable parameters: the position and pose of the camera, the pose includes the offset angle around the x and y axes, which is also represented as multiplication in code. According to the pose matrix, simulate the camera shooting angle, and project the three-dimensional model to a two-dimensional plane.

[0093] Specifically, set the camera's intrinsic parameters, the camera's intrinsic parameters are the IntrinsicsCamera function in the pyrender library, set to IntrinsicsCamera = (fx = 1600, fy = 1600, cx = w / 2, cy = h / 2), w, h are the length and width of the two-dimensional plane image.

[0094] Set the angles tilt_angle1 and rotation_angle1 to the angles of tilt around the x-axis and rotation around the y-axis respectively, convert them to radians, and calculate the tilt and rotation matrices.

[0095] The rotation matrix around the x-axis is:

[0096] tilt_matrix = np.array([

[0097] [1,0,0],

[0098] [0,np.cos(tilt_rad),-np.sin(tilt_rad)],

[0099] [0,np.sin(tilt_rad),np.cos(tilt_rad)]])

[0100] The rotation matrix around the y-axis is:

[0101] rotation_matrix=np.array([

[0102] [np.cos(rotation_rad),0,np.sin(rotation_rad)],

[0103] [0,1,0],

[0104] [-np.sin(rotation_rad),0,np.cos(rotation_rad)]])

[0105] Combine the rotation matrix R = rotation_matrix@tilt_matrix, expand and add the camera position matrix T to form the camera pose matrix RT.

[0106] Use the Mesh, Scene, and OffscreenRenderer functions of the pyrender library to project the 3D model onto a 2D plane and save the output. The projected image is shown in Figure 4 , Figure 5 . First, scene_mesh =

[0107] Mesh.from_trimesh(mesh), mesh: represents a 3D geometric model.

[0108] Mesh.from_trimesh(mesh): Converts the Trimesh object to the mesh object scene_mesh in the rendering library. Then creates a scene and sets the ambient light and background color: scene =

[0109] Scene(ambient_light=np.array([1.0, 1.0, 1.0, 1.0]), bg_color=[1, 0, 0, 1]), ambient_light: the ambient light color in the scene, bg_color: the background color of the scene. Add the mesh to the scene: scene.add(scene_mesh), add the camera to the scene:

[0110] scene.add(camera_intrinsics, pose=camera_pose), camera_intrinsics: the camera's intrinsic object, pose=camera_pose: the camera's pose matrix. Finally, create an off-screen renderer and render the scene: renderer=OffscreenRenderer(w, h), create an off-screen renderer with a viewport size of width w and height h, color_image, _ = renderer.render(scene), render the scene scene and return the projected image.

[0111] The third part, the projected images of the first and second parts are merged, and the position, time, and distance information are simulated, including: Step five.

[0112] Step five: After the two-dimensional image perspective transformation image is moderately cut, it is merged with the three-dimensional projection to the two-dimensional plane image to form an image simulating the shooting of the sea ice area on the sea surface, and then the corresponding simulation information is added.

[0113] Specifically, after the two-dimensional image perspective transformation image is moderately cut, see Figure 3 , it is merged with the three-dimensional projection to the two-dimensional plane image, and the merged picture is seen in Figure 6 and Figure 7 .

[0114] First, ensure that the size of the three-dimensional model projected to the two-dimensional plane image is the same as the background image, which requires setting w and h in renderer=OffscreenRenderer(w, h) to be equal to the length and width of the background image.

[0115] Then, create a mask to find all black pixel points in the three-dimensional projection to the two-dimensional plane image, and fill the black area of the moderately cut two-dimensional image to get a new image after merging the two images.

[0116] Simulate the shooting position, time, and distance of the shooting position and the sea ice position of the merged sea ice image.

[0117] By the method for generating a sea ice area on a sea surface through simulation, effective data sets can be obtained, and subsequent deep learning or machine learning training is facilitated.

[0118] Obviously, the above embodiments are only examples for clearly illustrating the present application, and are not intended to limit the embodiments. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments are not required to be exhausted, and the obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A method for simulating and generating sea ice areas on the sea surface, characterized in that: The following steps are involved: Part I: Perspective projection of two-dimensional sea surface images; Part II: Projection of a 3D model onto a 2D plane; Part 3: Merge the projected images of the first and second parts and add the corresponding simulation information; The first part includes steps one and two: Step 1: Simulate the shooting state of the camera, set the internal and external parameters of the camera model, simulate the ideal camera model state, and set the adjustable parameters: focal length, tilt and rotation angle; Step 2: Perform perspective transformation on the photographed sea surface; adjust the tilt and rotation angles, generate the tilt matrix R_x around the x-axis and the rotation matrix R_z around the z-axis according to the adjusted angles, multiply the two matrices to output a combined rotation matrix R, and apply the combined rotation matrix, camera parameter matrix, and distortion parameter matrix as parameters to the cv2.initUndistortRectifyMap function to output two mapping matrices mapx and mapy. These two mapping matrices are passed as parameters to the cv2.remap function to output the result of the perspective transformation, simulating the effect that objects appear larger when near and smaller when far away. Among the adjustable parameters in step 1, the tilt angle is the angle of transformation along the x-axis, and the rotation angle is the angle of transformation along the z-axis; The perspective transformation function in step 2 includes: Initialize the mapping coordinates and apply the cv2.initUndistortRectifyMap function to return two mapping matrices mapx and mapy; Remap the image and apply the cv2.remap function to remap the input distorted image to the corrected image according to mapx and mapy; The second part includes steps three and four: Step 3: Find a 3D model to render and simulate icebergs and floating ice. Use a rockery model to render and simulate icebergs, and use a lawn to simulate floating ice. Step 4: Project the rendered 3D model onto a 2D plane. Set the camera's intrinsic and extrinsic parameters, and then set the adjustable parameters: the camera's position and attitude. The attitude includes the offset angles around the x and y axes, which are also expressed as multiplications in the code. Based on the pose matrix, simulate the camera's shooting angle and project the 3D model onto a 2D plane.

2. The method for simulating and generating sea ice areas on the sea surface according to claim 1, characterized in that: In step 1, the intrinsic parameters of the camera model are: [[focal_length,0,cx], [0,focal_length,cy], [0,0,1]] Among them, focal_length is the focal length of the camera model, cx and cy are the center positions of the sea surface photo respectively; The tilt matrix around the x-axis is: R_x=np.array([ [1,0,0], [0,np.cos(tilt_rad),-np.sin(tilt_rad)], [0,np.sin(tilt_rad),np.cos(tilt_rad)]]) The rotation matrix around the z-axis is: R_z=np.array([ [np.cos(rotation_rad),-np.sin(rotation_rad),0], [np.sin(rotation_rad),np.cos(rotation_rad),0], [0,0,1]])； The combined rotation matrix R satisfies: R = R_x@R_z.

3. The method for simulating and generating sea ice areas on the sea surface according to claim 1, characterized in that: In step 4, the intrinsic parameters of the camera are the IntrinsicsCamera function in the pyrender library, which is set to IntrinsicsCamera = (fx = 1600, fy = 1600, cx = w / 2, cy = h / 2), where w and h are the length and width of the two-dimensional image. The tilt_angle and rotation_angle of the 3D model projected onto the 2D plane are the angles of tilt around the x-axis and rotation around the y-axis, which are converted into radians to calculate the tilt and rotation matrices. The rotation matrix around the x-axis is: tilt_matrix = np.array([ [1,0,0], [0,np.cos(tilt_rad),-np.sin(tilt_rad)], [0,np.sin(tilt_rad),np.cos(tilt_rad)]]) The rotation matrix around the y-axis is: rotation_matrix=np.array([ [np.cos(rotation_rad),0,np.sin(rotation_rad)], [0,1,0], [-np.sin(rotation_rad),0,np.cos(rotation_rad)]]) Combine the rotation matrix expansion and the camera position matrix to form the camera pose matrix; After the camera's pose matrix is ​​set, use the Mesh, Scene, and OffscreenRenderer functions of the pyrender library to project the 3D model onto a 2D plane and save the output.

4. The method for simulating and generating sea ice areas on the sea surface according to claim 1, characterized in that: Part III includes: Step 5; Step 5: The perspective-transformed 2D image is appropriately sheared and merged with the image of the 3D model projected onto the 2D plane to form an image simulating the sea ice area, and then the corresponding simulation information is added.

5. The method for simulating and generating sea ice areas on the sea surface according to claim 4, characterized in that: The corresponding simulation information includes: location, time and distance information.

Images