Underwater VR image processing method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides an underwater VR image processing method, comprising: obtaining the underwater environment information, the first underwater image and the second underwater image; generating a third underwater image according to the underwater environment information, the first underwater image and the second underwater image; generating a target VR image according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space, and the target VR image is used to be displayed in the image display unit. The present invention can avoid the direct influence of underwater factors, thereby reducing the difficulty of virtualization production, reducing the image distortion of the underwater VR device, and improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to an underwater VR image processing method, device, electronic equipment and storage medium. Background Art

[0002] Virtual reality technology (English name: Virtual Reality, abbreviated as VR) is a new practical technology developed in the 20th century. Virtual reality technology includes computers, electronic information, and simulation technology. Its basic implementation method is to use computers to simulate virtual environments to give people a sense of environmental immersion.

[0003] Current virtual production methods usually require establishing real shooting scenes in real scenes, and realizing virtual shooting through technologies such as optical tracking of cameras and linkage of virtual cameras in virtual scenes. This method has great limitations for underwater shooting and virtual production. For example, the position of the shooting equipment in underwater coordinates, movement trajectory, lighting, and camera operation methods have a great impact on the underwater shooting effect, making underwater virtual production more difficult, which in turn leads to high image distortion of VR devices, resulting in a poor user experience. Summary of the invention

[0004] The embodiment of the present invention provides an underwater VR image processing method, which aims to solve the problem that in the existing virtualization production, the underwater factors have a great influence on the underwater shooting effect, which makes the underwater virtualization production difficult, and further leads to a high image distortion of the underwater VR device. The method generates a third underwater image by acquiring the underwater environment information of the area where the current device is located, a first underwater image corresponding to the first area, and a second underwater image corresponding to the second area, and then generates a target VR image for display by the third underwater image and a preset three-dimensional underwater space. Since the third underwater image is an image generated through multiple dimensions, the third underwater image has more accurate environment and image information, thereby avoiding the direct influence of underwater factors, thereby reducing the difficulty of virtualization production, reducing the image distortion of the underwater VR device, and improving user experience.

[0005] In a first aspect, an embodiment of the present invention provides an underwater VR image processing method, which is applied to an underwater VR device, wherein the underwater VR device includes a waterproof housing, a processor unit, an image display unit, an underwater environment sensing unit, a first image acquisition unit, and a second image acquisition unit arranged in the waterproof housing, wherein the underwater environment sensing unit is used to acquire underwater environment information of an area where the underwater VR device is located, the first image acquisition unit is used to acquire a first underwater image of a first area, and the second image acquisition unit is used to acquire a second underwater image of a second area, wherein an intersection-over-union ratio between the first area and the second area is less than a preset value, and the method includes the following steps:

[0006] Acquiring the underwater environment information, the first underwater image, and the second underwater image;

[0007] generating a third underwater image according to the underwater environment information, the first underwater image, and the second underwater image;

[0008] A target VR image is generated based on the third underwater image and a preset three-dimensional underwater space. The three-dimensional underwater space and the area where the VR device is located have the same activity space. The target VR image is used to be displayed in the image display unit.

[0009] Optionally, the underwater environment sensing unit includes at least one of an underwater depth sensor, an underwater pressure sensor, an underwater light sensor, an underwater temperature sensor, and a water quality transparency detector, the underwater environment information includes at least one underwater environment information such as underwater depth information, underwater pressure information, underwater light information, underwater temperature information, and water quality transparency information, and generating a third underwater image according to the underwater environment information, the first underwater image, and the second underwater image includes:

[0010] encoding and quantizing the underwater environment information using a pixel value range of the first underwater image or the second underwater image to obtain a noise map;

[0011] The noise map, the first underwater image and the second underwater image are processed by a preset image generation network to generate the third underwater image.

[0012] Optionally, the preset image generation network includes a first image generation network, a second image generation network, and a third image generation network, and performing image generation processing on the noise map, the first underwater image, and the second underwater image through the preset generation network to generate the third underwater image includes:

[0013] Performing image generation processing on the noise map and the first underwater image through the first image generation network to obtain a first generated image, and performing image generation processing on the noise map and the second underwater image through the second image generation network to obtain a second generated image;

[0014] The first generated image and the second generated image are processed by the third image generation network to obtain a third underwater image.

[0015] Optionally, generating a target VR image according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space, includes:

[0016] adjusting the preset three-dimensional underwater space according to the third underwater image to obtain an adjusted three-dimensional underwater space;

[0017] calculating virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space according to the third underwater image;

[0018] A target VR image is generated according to the adjusted virtual shooting parameters of the three-dimensional underwater space.

[0019] Optionally, the underwater VR device includes a master device and a slave device, the master device includes a first waterproof shell, a processor unit, an image display unit, an underwater environment sensing unit, and a first image acquisition unit arranged in the first waterproof shell, the slave device includes a second waterproof shell, and a second image acquisition unit arranged in the second waterproof shell, the master device is worn on a first specific part of the human body, the slave device is worn on a second specific part of the human body, the second image acquisition unit is connected to the processor unit by signal, and the virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space are calculated according to the third underwater image, including:

[0020] Determining a user visual parameter of the underwater VR device in a current state according to a size parameter of the third underwater image and an intersection-and-union ratio of the first underwater image and the second underwater image;

[0021] The virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space are calculated according to the user visual parameters.

[0022] Optionally, determining the user visual parameter of the underwater VR device in the current state according to the size parameter of the third underwater image and the intersection-and-union ratio of the first underwater image and the second underwater image includes:

[0023] Calculating a user posture of the underwater VR device in a current state according to a size parameter of the third underwater image and an intersection-and-union ratio of the first underwater image and the second underwater image;

[0024] According to the user posture, the user visual parameters of the underwater VR device in the current state are determined.

[0025] Optionally, after the step of generating a target VR image according to the third underwater image and a preset three-dimensional underwater space, the method includes:

[0026] Calculate the color difference pixels appearing in the target VR image;

[0027] Analyze the color difference of the color difference pixel point;

[0028] Performing color difference compensation on the color difference pixel points with the opposite color of the color difference to obtain an image to be displayed;

[0029] The image to be displayed is transmitted to the image display unit for display.

[0030] In a second aspect, an embodiment of the present invention provides an underwater VR image processing device, which is applied to an underwater VR device. The underwater VR device includes a waterproof housing, an underwater VR image processing device, an image display unit, an underwater environment sensing unit, a first image acquisition unit, and a second image acquisition unit arranged in the waterproof housing. The underwater environment sensing unit is used to collect underwater environment information of an area where the underwater VR device is located. The first image acquisition unit is used to collect a first underwater image of a first area. The second image acquisition unit is used to collect a second underwater image of a second area. The intersection-and-union ratio of the first area to the second area is less than a preset value. The device includes:

[0031] An acquisition module, used to acquire the underwater environment information, the first underwater image and the second underwater image;

[0032] A first generating module, configured to generate a third underwater image according to the underwater environment information, the first underwater image and the second underwater image;

[0033] The second generating module is used to generate a target VR image according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the underwater VR image processing method provided in the embodiment of the present invention are implemented.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the underwater VR image processing method provided in the embodiment of the invention are implemented.

[0036] In an embodiment of the present invention, the underwater environment information, the first underwater image, and the second underwater image are obtained; a third underwater image is generated according to the underwater environment information, the first underwater image, and the second underwater image; a target VR image is generated according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space, and the target VR image is used to be displayed in the image display unit. The third underwater image is generated by obtaining the underwater environment information of the area where the current device is located, the first underwater image corresponding to the first area, and the second underwater image corresponding to the second area, and then the target VR image is generated by the third underwater image and the preset three-dimensional underwater space for display. Since the third underwater image is an image generated through multiple dimensions, the third underwater image has more accurate environment and image information, thereby avoiding the direct influence of underwater factors, thereby reducing the difficulty of virtualization production, reducing the image distortion of the underwater VR device, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a structural diagram of an underwater VR device provided by an embodiment of the present invention;

[0039] Figure 2 It is a flow chart of an underwater VR image processing method provided by an embodiment of the present invention;

[0040] Figure 3 It is a structural schematic diagram of an underwater VR image processing device provided by an embodiment of the present invention;

[0041] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] See also Figure 1 , Figure 1 is a structural diagram of an underwater VR device provided by an embodiment of the present invention, such as Figure 1 As shown, the underwater VR device includes a waterproof shell, a processor unit, an image display unit, an underwater environment sensing unit, a first image acquisition unit and a second image acquisition unit arranged in the waterproof shell, the underwater environment sensing unit is used to collect underwater environment information of the area where the underwater VR device is located, the first image acquisition unit is used to collect a first underwater image of the first area, and the second image acquisition unit is used to collect a second underwater image of the second area, and the intersection-and-union ratio of the first area and the second area is less than a preset value.

[0044] Among them, the processor unit is respectively connected with the image display unit, the underwater environment sensing unit, the first image acquisition unit and the second image acquisition unit for signal connection. After the underwater environment sensing unit transmits the collected underwater environment information to the processor unit, the first image acquisition unit transmits the collected first underwater image to the processor unit, and the second image acquisition unit transmits the collected second underwater image to the processor unit, the processor unit generates a third underwater image according to the underwater environment information, the first underwater image and the second underwater image; generates a target VR image according to the third underwater image and a preset three-dimensional underwater space. The three-dimensional underwater space and the area where the VR device is located have the same activity space, and the target VR image is used to be displayed in the image display unit.

[0045] The intersection-and-union ratio of the first area and the second area is smaller than a preset value, thereby ensuring that the first underwater image and the second underwater image do not overlap.

[0046] The signal connection may be any existing wired or wireless signal connection mode. In the embodiment of the present invention, considering the signal stability in the underwater environment, the signal connection may be a wired connection.

[0047] Specifically, an image generation network is embedded in the processor unit. After receiving the underwater environment information, the first underwater image and the second underwater image, the processor unit performs image generation processing on the underwater environment information, the first underwater image and the second underwater image through the image generation network to obtain a third underwater image.

[0048] Furthermore, the processor unit stores a three-dimensional underwater space, which can be obtained by pre-real scene modeling based on the current underwater space, and by visual art adjustment based on the model obtained by real scene modeling. The visual art adjustment can be understood as a stylized model adjustment. For example, the current underwater space is the underwater space of a swimming pool. After real scene modeling of the underwater space of the swimming pool, an underwater space model of the swimming pool can be obtained. After visual art adjustment of the underwater space module of the swimming pool, an underwater space model of a certain ocean, an underwater space model of a certain lake, and an underwater space model of a certain river can be obtained. Among them, the preset three-dimensional underwater space and the area where the VR device is located have the same activity space, so as to avoid collision between the user and the inactive space during the experience.

[0049] After obtaining the third underwater image, since the third underwater image is an image of the actual underwater space, the processor unit can generate a target VR image corresponding to the third underwater image in the preset three-dimensional underwater space, so that the above target VR image has the same style as the above preset three-dimensional underwater space.

[0050] After obtaining the target VR image, the processor unit can transmit the target VR image to the image display unit for display.

[0051] See also Figure 2 , Figure 2 is a flow chart of an underwater VR image processing method provided by an embodiment of the present invention. Figure 2 As shown, the underwater VR image processing method is mainly applied to the above-mentioned processor unit, and includes the following steps:

[0052] 201. Acquire underwater environment information, a first underwater image, and a second underwater image.

[0053] In the embodiment of the present invention, the underwater environment information can be collected in real time by the underwater environment sensing unit, and the underwater environment sensing unit transmits the underwater environment information collected in real time to the processor unit.

[0054] The first underwater image can be acquired in real time by the first image acquisition unit, and the first image acquisition unit transmits the first underwater image acquired in real time to the processor unit.

[0055] The second underwater image can be acquired in real time by the second image acquisition unit, and the second image acquisition unit transmits the second underwater image acquired in real time to the processor unit.

[0056] The above-mentioned underwater environmental information may include at least one of underwater depth information, underwater pressure information, underwater light information, underwater temperature information, water quality transparency and other environmental information.

[0057] The first underwater image is obtained by real-time acquisition of the first area by the first image acquisition unit, and the second underwater image is obtained by real-time acquisition of the second area by the second image acquisition unit. The first area is understood to be the field of view of the first image acquisition unit, and the second area is the field of view of the second image acquisition unit. For example, the first area may be the area directly in front of the top of the user's head, and the second area may be the lower front area of ​​the user's chest.

[0058] 202. Generate a third underwater image according to the underwater environment information, the first underwater image, and the second underwater image.

[0059] In an embodiment of the present invention, the above-mentioned underwater environmental information may include at least one of the environmental information such as underwater depth information, underwater pressure information, underwater light information, underwater temperature information, water quality transparency information, etc., and the third underwater image can be generated based on at least one of the environmental information such as underwater depth information, underwater pressure information, underwater light information, underwater temperature information, water quality transparency, etc., in combination with the first underwater image and the second underwater image.

[0060] In a possible embodiment, in the process of generating the third underwater image through the above-mentioned underwater environmental information, the first underwater image and the second underwater image, it can be understood that the first underwater image and the second underwater image are filtered in real time through the underwater environmental information, and the first underwater image after real-time filtering is spliced ​​with the second underwater image to obtain the third underwater image.

[0061] The size of the third underwater image is larger than that of the first underwater image and the second underwater image, and the sizes of the first underwater image and the second underwater image are fixed sizes, which are related to the setting parameters of the first image acquisition unit and the second image acquisition unit. In an embodiment of the present invention, the size of the first underwater image is the same as that of the second underwater image.

[0062] The third underwater image includes an image area corresponding to the first area and an image area corresponding to the second area.

[0063] 203. Generate a target VR image according to the third underwater image and the preset three-dimensional underwater space.

[0064] In an embodiment of the present invention, the three-dimensional underwater space and the area where the VR device is located have the same activity space, and the target VR image is used to be displayed in the image display unit.

[0065] The above-mentioned three-dimensional underwater space can be obtained by pre-real scene modeling based on the current underwater space, and visual art adjustment can be made based on the model obtained by real scene modeling. The above-mentioned visual art adjustment can be understood as a stylized model adjustment. For example, the current underwater space is the underwater space of a swimming pool. After real scene modeling of the underwater space of the swimming pool, an underwater space model of the swimming pool can be obtained. After visual art adjustment of the underwater space module of the swimming pool, an underwater space model of a certain ocean, an underwater space model of a certain lake, and an underwater space model of a certain river can be obtained. Among them, the preset three-dimensional underwater space and the area where the VR device is located have the same activity space to avoid collision between users and inactive spaces during the experience.

[0066] Specifically, the target VR image can be obtained based on the third underwater image and an image corresponding to the third underwater image in a preset three-dimensional underwater space area.

[0067] More specifically, the real-scene feature points in the current underwater space can be mapped one-to-one with the stylized feature points in the three-dimensional underwater space, the positions of the real-scene feature points in the third underwater image can be calculated, the positions of the stylized feature points can be determined based on the positions of the real-scene feature points, and then the corresponding candidate VR image can be obtained in the three-dimensional underwater space based on the positions of the stylized feature points. The size of the candidate VR image is the same as that of the third underwater image, and the corresponding target VR image is cropped from the middle part of the candidate VR image.

[0068] It should be noted that the size of the above-mentioned target VR image is the same as the display size of the image display unit.

[0069] In an embodiment of the present invention, the underwater environment information, the first underwater image, and the second underwater image are obtained; a third underwater image is generated according to the underwater environment information, the first underwater image, and the second underwater image; a target VR image is generated according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space, and the target VR image is used to be displayed in the image display unit. The third underwater image is generated by obtaining the underwater environment information of the area where the current device is located, the first underwater image corresponding to the first area, and the second underwater image corresponding to the second area, and then the target VR image is generated by the third underwater image and the preset three-dimensional underwater space for display. Since the third underwater image is an image generated through multiple dimensions, the third underwater image has more accurate environment and image information, thereby avoiding the direct influence of underwater factors, thereby reducing the difficulty of virtualization production, reducing the image distortion of the underwater VR device, and improving the user experience.

[0070] Optionally, the underwater environment sensing unit includes at least one of an underwater depth sensor, an underwater pressure sensor, an underwater light sensor, an underwater temperature sensor, and a water quality transparency detector, and the underwater environment information includes at least one of underwater depth information, underwater pressure information, underwater light information, underwater temperature information, water quality transparency information, etc. In the step of generating a third underwater image based on the underwater environment information, the first underwater image, and the second underwater image, the underwater environment information can be encoded and quantized within the pixel value range of the first underwater image or the second underwater image to obtain a noise map; the noise map, the first underwater image, and the second underwater image are subjected to image generation processing through a preset image generation network to generate a third underwater image.

[0071] In the embodiment of the present invention, the range of coding and quantization can be determined according to the pixel value range of the first underwater image or the pixel value range of the second underwater image. In a possible embodiment, the pixel value range of the first underwater image can be an integer interval of [0, 255], and the pixel value range of the second underwater image can also be an integer interval of [0, 255]. After converting the underwater environmental information collected by the sensor into digital form, the underwater environmental information in digital form is obtained, and the underwater environmental information in digital form is coded and quantized to an integer interval of [0, 255].

[0072] Specifically, the underwater environment information in digital mode can be encoded by word vector encoding. In word vector encoding, a character can be represented by a multidimensional vector. For example, 1 can be represented by an n-dimensional vector [a0, a1, ..., an], where n is the total number of pixels in the first underwater image or the total number of pixels in the second underwater image. Since the size of the first underwater image is the same as the size of the second underwater image, the total number of pixels in the first underwater image or the total number of pixels in the second underwater image is the same. After encoding the underwater environment information by word vector encoding, the encoding vector of the underwater environment information is obtained, and then the encoding vector is mapped to an integer interval of [0, 255] for quantization to obtain a noise map.

[0073] By encoding and quantizing the underwater environment information, a noise image having the same size as the first underwater image and the second underwater image can be obtained.

[0074] The preset image generation network can be based on the generator in the generative adversarial network, and the generative adversarial network can use noise information as input to generate corresponding image information. In an embodiment of the present invention, when the image generation network is trained, the training sample is a triple structure, and the training sample includes a sample noise map, a sample first underwater image, and a sample second underwater image. Each training sample also corresponds to a repaired label image, wherein the acquisition method of the sample noise map is the same as the acquisition method of the noise map, and the acquisition method of the sample first underwater image and the sample second underwater image is the same as the acquisition method of the first underwater image and the second underwater image. The repaired label image can be an image that is manually repaired, and the repaired image is used to identify the generated image of the generator in the discriminator. Through the adversarial training of the generator and the discriminator, the generator learns the ability to generate images close to the repaired label image. After training, only the parameters of the generator are deployed in the processor unit, and the training samples are used in the processor unit to fine-tune the parameters of the generator, thereby obtaining an image generation network.

[0075] By generating the third underwater image through a preset image generation network, a third underwater image that is less affected by the underwater environment can be obtained.

[0076] Optionally, the preset image generation network includes a first image generation network, a second image generation network, and a third image generation network. In the step of generating and processing the noise map, the first underwater image and the second underwater image through the preset generation network to generate the third underwater image, the noise map and the first underwater image can be image generated and processed by the first image generation network to obtain a first generated image, and the noise map and the second underwater image can be image generated and processed by the second image generation network to obtain a second generated image; and the first generated image and the second generated image can be image generated and processed by the third image generation network to obtain a third underwater image.

[0077] In the embodiment of the present invention, the noise map and the first underwater image are processed by the first image generation network to generate the first generated image, which can make the first generated image closer to the restored image; the noise map and the second underwater image are processed by the second image generation network to generate the second generated image, which can make the second generated image closer to the restored image; further, in the processing of the third image generation network, the first generated image and the second generated image are processed to generate the third underwater image, which makes the third underwater image closer to the restored image. Reduce image distortion, make the image clearer, and improve user experience.

[0078] The output of the first image generation network and the output of the second image generation network are both connected to the input of the third image generation network.

[0079] It should be noted that since the image generation network includes the first image generation network, the second image generation network and the third image generation network, it is necessary to construct the first generator, the second generator and the third generator during training, wherein the first generator corresponds to the first image generation network, the second generator corresponds to the second image generation network, and the third generator corresponds to the third image generation network. It is also necessary to construct the first discriminator, the second discriminator and the third discriminator, wherein the output of the first generator is respectively connected to the input of the first discriminator and the third generator, the output of the second generator is respectively connected to the input of the second discriminator and the third generator, and the output of the third generator is connected to the input of the third discriminator.

[0080] During training, the training samples are triplet structures, including sample noise images, sample first underwater images, and sample second underwater images. Each training sample also corresponds to a repaired label image triplet, which includes a repaired first label image, a repaired second label image, and a repaired third label image. The repaired first label image is used for identification by the first discriminator, the repaired second label image is used for identification by the second discriminator, and the third label image is used for identification by the third discriminator. The acquisition method of the above-mentioned sample noise image is the same as the acquisition method of the above-mentioned noise image, and the acquisition methods of the sample first underwater image and the sample second underwater image are the same as the acquisition methods of the above-mentioned first underwater image and the above-mentioned second underwater image. The repaired label image can be an image repaired manually, and the repaired image is used to identify the generated image of the corresponding generator in the corresponding discriminator. Through the adversarial training of the generator and the discriminator, the generator learns the ability to generate images close to the repaired label image. After training, only the parameters of the generator are deployed in the processor unit, and the training samples are used in the processor unit to fine-tune the parameters of the generator, thereby obtaining an image generation network. During the training process, the total loss of the first generator, the second generator, and the third generator can be used to perform error backpropagation on the first generator, the second generator, and the third generator to adjust the parameters of the first generator, the second generator, and the third generator.

[0081] Optionally, a target VR image is generated based on the third underwater image and a preset three-dimensional underwater space, and in the step that the three-dimensional underwater space and the area where the VR device is located have the same activity space, the preset three-dimensional underwater space can be adjusted according to the third underwater image to obtain an adjusted three-dimensional underwater space; according to the third underwater image, virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space are calculated; according to the virtual shooting parameters of the adjusted three-dimensional underwater space, a target VR image is generated.

[0082] In the embodiment of the present invention, the activity space information in the third underwater image can be extracted to adjust the activity space in the three-dimensional underwater space to obtain the adjusted three-dimensional underwater space. The above-mentioned activity space refers to the space where the user can move in the water. For example, when another user appears in the third underwater image, the space where the other user is located is not the space where the current user can move at the current moment. In this way, the three-dimensional underwater space can be adjusted in real time, thereby avoiding unintentional collisions between multiple users and protecting users and underwater VR equipment.

[0083] The virtual shooting parameters include parameters such as the shooting focal length, shooting position and shooting direction of the virtual camera. Through the virtual shooting parameters, an image having the same shooting focal length, shooting position and shooting direction as the third underwater image can be obtained, and the target VR image can be obtained based on the image.

[0084] Optionally, the underwater VR device includes a master device and a slave device, the master device includes a first waterproof shell, a processor unit, an image display unit, an underwater environment sensing unit, and a first image acquisition unit arranged in the first waterproof shell, the slave device includes a second waterproof shell, and a second image acquisition unit arranged in the second waterproof shell, the master device is worn on a first specific part of the human body, the slave device is worn on a second specific part of the human body, and the second image acquisition unit is signal-connected to the processor unit. In the step of calculating the virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space according to the third underwater image, the user visual parameters of the underwater VR device in the current state can be determined according to the size parameters of the third underwater image and the intersection and union ratio of the first underwater image and the second underwater image; and the virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space are calculated according to the user visual parameters.

[0085] In an embodiment of the present invention, the user visual parameter includes the user's visual range based on the current posture, for example, there is a mapping relationship between the user visual parameter and the virtual shooting parameter. Further, the user visual parameter can be represented by the user's posture, and further, the user visual parameter can be represented by the user's head state.

[0086] The above-mentioned underwater VR device includes a master device and a slave device, and the above-mentioned master device and the slave device are worn on different parts of the human body, so that different first underwater images and second underwater images can be collected. For example, the master device can be worn on the head of the human body, and the slave device can be worn on the trunk or limbs of the human body. In the embodiment of the present invention, the master device is worn on the first specific part of the human body, and the slave device is worn on the second specific part of the human body. The first specific part of the human body is the head, and the above-mentioned second specific part of the human body is the chest or shoulder. In this way, the posture information between the user's head and chest or shoulder can be obtained through the first underwater image and the second underwater image, so that the third image also has the posture information between the user's head and chest or shoulder. For example, when a person is standing upright and looking straight ahead, the first image acquisition unit and the second image acquisition unit can respectively collect images of the first area and the second area in front of the human body. When the person lowers his head, the overlapping area of ​​the first area and the second area becomes larger, and when the person raises his head, the overlapping area of ​​the first area and the second area becomes smaller.

[0087] It should be noted that the intersection ratio of the first area and the second area is less than a preset value. It can be understood that when the user is standing upright and looking straight ahead, the ratio of the intersection area and the merged area of ​​the first area in front of the user is less than the preset value, thereby ensuring that the first area and the second area will not completely overlap when the user is standing upright and looking straight ahead.

[0088] Of course, in a possible embodiment, the intersection-and-union ratio of the first area and the second area is less than the first preset value and greater than the second preset value, the first preset value is 1, and the second preset value is 0, so that there is a certain area overlap between the first area and the second area, but there is no complete overlap.

[0089] The third underwater image includes an image area corresponding to the first underwater image and an image area corresponding to the second underwater image. The image area corresponding to the first underwater image is an image corresponding to the first area in the third underwater image, and the image area corresponding to the second underwater image is an image corresponding to the second area in the third underwater image. Thus, the user's posture can be obtained according to the size of the third underwater image. When the size of the third underwater image is smaller than the standard size, it indicates that the user is in a head-down posture. When the size of the third underwater image is larger than the standard size, it indicates that the user is in a head-up posture. The standard size of the third underwater image is the size of the third underwater image generated according to the first underwater image and the second underwater image when the user is standing upright and looking straight ahead.

[0090] Optionally, in the step of determining the user visual parameters of the underwater VR device in the current state according to the size parameters of the third underwater image and the intersection and union ratio of the first underwater image and the second underwater image, the user posture of the underwater VR device in the current state can be calculated according to the size parameters of the third underwater image and the intersection and union ratio of the first underwater image and the second underwater image; and the user visual parameters of the underwater VR device in the current state can be determined according to the user posture.

[0091] In an embodiment of the present invention, when the size of the third underwater image is smaller than the standard size, it indicates that the user is in a head-down posture, and the smaller the size of the third underwater image, the greater the head-down amplitude; when the size of the third underwater image is larger than the standard size, it indicates that the user is in a head-up posture, and the larger the size of the third underwater image, the greater the head-up amplitude.

[0092] At the same time, when the intersection-and-union ratio of the first underwater image and the second underwater image is greater than the standard intersection-and-union ratio, it means that the user is in a head-down posture. The larger the intersection-and-union ratio of the first underwater image and the second underwater image, the greater the head-down amplitude. When the intersection-and-union ratio of the first underwater image and the second underwater image is less than the standard intersection-and-union ratio, it means that the user is in a head-up posture, where the standard intersection-and-union ratio is the intersection-and-union ratio of the first underwater image and the second underwater image when the user is standing upright and looking straight ahead.

[0093] The user posture of the underwater VR device in the current state can be determined based on the size parameter of the third underwater image and the intersection-and-combination ratio of the first underwater image and the second underwater image. Specifically, the user state is the head posture, and the user's head posture can be determined by the following formula:

[0094]

[0095] s1-s0>0, take +

[0096] s1-s0<0, take -

[0097] Wherein, σ is the user's head posture angle, positive means raising the head, negative means lowering the head, σ0 is the standard posture angle, which can be measured by an angle measuring instrument when the user is standing upright and looking straight ahead, s0 is the standard size of the third underwater image, s1 is the current size of the third underwater image, a0∩b0 / a0∪b0 is the standard intersection-and-union ratio of the first underwater image and the second underwater image, and a1∩b1 / a1∪b1 is the current intersection-and-union ratio of the first underwater image and the second underwater image.

[0098] According to the user's head posture angle, the user visual parameters of the underwater VR device in the current state are obtained.

[0099] Optionally, after the step of generating a target VR image based on the third underwater image and the preset three-dimensional underwater space, the chromatic aberration pixels appearing in the target VR image can also be calculated; the chromatic aberration colors of the chromatic aberration pixels can be analyzed; the chromatic aberration of the chromatic aberration pixels can be compensated with the opposite colors of the chromatic aberration colors to obtain an image to be displayed; and the image to be displayed can be transmitted to an image display unit for display.

[0100] In an embodiment of the present invention, the above-mentioned chromatic aberration pixel points are generated due to the refractive index of the material. White light can be dispersed into light of other colors through refraction. Therefore, the chromatic aberration of the chromatic aberration pixel points can be compensated by using opposite colors to obtain an image to be displayed with higher image quality. The image to be displayed is then transported to the image display unit for display, thereby realizing the virtual reality function of the underwater VR device.

[0101] In implementing the embodiment of the present invention, a third underwater image is generated by acquiring underwater environmental information of the area where the current device is located, a first underwater image corresponding to the first area, and a second underwater image corresponding to the second area. Then, a target VR image is generated from the third underwater image and a preset three-dimensional underwater space for display. Since the third underwater image is an image generated through multiple dimensions, the third underwater image has more accurate environmental and image information, thereby avoiding the direct influence of underwater factors, thereby reducing the difficulty of virtualization production, reducing the image distortion of underwater VR equipment, and improving user experience.

[0102] It should be noted that the underwater VR image processing method provided in the embodiment of the present invention can be applied to VR devices, smart phones, computers, servers and other devices.

[0103] Optionally, an embodiment of the present invention provides an underwater VR image processing device, which is applied to an underwater VR device. The underwater VR device includes a waterproof housing, an underwater VR image processing device, an image display unit, an underwater environment sensing unit, a first image acquisition unit, and a second image acquisition unit arranged in the waterproof housing. The underwater environment sensing unit is used to collect underwater environment information of an area where the underwater VR device is located. The first image acquisition unit is used to collect a first underwater image of a first area. The second image acquisition unit is used to collect a second underwater image of a second area. The intersection-and-union ratio of the first area to the second area is less than a preset value. See Figure 3 , Figure 3 is a structural schematic diagram of an underwater VR image processing device provided by an embodiment of the present invention, such as Figure 3 As shown, the device comprises:

[0104] An acquisition module 301 is used to acquire the underwater environment information, the first underwater image and the second underwater image;

[0105] A first generating module 302, configured to generate a third underwater image according to the underwater environment information, the first underwater image and the second underwater image;

[0106] The second generating module 303 is used to generate a target VR image according to the third underwater image and a preset three-dimensional underwater space, where the three-dimensional underwater space and the area where the VR device is located have the same activity space.

[0107] Optionally, the underwater environment sensing unit includes at least one of an underwater depth sensor, an underwater pressure sensor, an underwater light sensor, an underwater temperature sensor and a water quality transparency detector, and the underwater environment information includes at least one underwater environment information such as underwater depth information, underwater pressure information, underwater light information, underwater temperature information, water quality transparency information, etc. The first generation module 302 is also used to encode and quantize the underwater environment information within the pixel value range of the first underwater image or the second underwater image to obtain a noise map; the noise map, the first underwater image and the second underwater image are subjected to image generation processing through a preset image generation network to generate the third underwater image.

[0108] Optionally, the preset image generation network includes a first image generation network, a second image generation network, and a third image generation network, and the first generation module 302 is also used to perform image generation processing on the noise map and the first underwater image through the first image generation network to obtain a first generated image, and to perform image generation processing on the noise map and the second underwater image through the second image generation network to obtain a second generated image; and to perform image generation processing on the first generated image and the second generated image through the third image generation network to obtain a third underwater image.

[0109] Optionally, the second generating module 303 is also used to adjust the preset three-dimensional underwater space according to the third underwater image to obtain an adjusted three-dimensional underwater space; calculate the virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space according to the third underwater image; and generate a target VR image according to the virtual shooting parameters of the adjusted three-dimensional underwater space.

[0110] Optionally, the underwater VR device includes a master device and a slave device, the master device includes a first waterproof shell, a processor unit, an image display unit, an underwater environment sensing unit, and a first image acquisition unit arranged in the first waterproof shell, the slave device includes a second waterproof shell, and a second image acquisition unit arranged in the second waterproof shell, the master device is worn on a first specific part of the human body, the slave device is worn on a second specific part of the human body, the second image acquisition unit is signal-connected to the processor unit, and the second generation module 303 is also used to determine the user visual parameters of the underwater VR device in the current state according to the size parameters of the third underwater image and the intersection and union ratio of the first underwater image and the second underwater image; and calculate the virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space according to the user visual parameters.

[0111] Optionally, the second generating module 303 is also used to calculate the user posture of the underwater VR device in the current state according to the size parameters of the third underwater image and the intersection and union ratio of the first underwater image and the second underwater image; and determine the user visual parameters of the underwater VR device in the current state according to the user posture.

[0112] Optionally, the device comprises:

[0113] A calculation module, used to calculate the color difference pixels appearing in the target VR image;

[0114] A parsing module, used for parsing the color difference of the color difference pixel point;

[0115] A compensation module, used for performing color difference compensation on the color difference pixel points with the opposite color of the color difference to obtain an image to be displayed;

[0116] The display module is used to transmit the image to be displayed to the image display unit for display.

[0117] It should be noted that the behavior detection device provided in the embodiment of the present invention can be applied to VR devices, smart phones, computers, servers and other devices that can perform underwater VR image processing.

[0118] The behavior detection device provided in the embodiment of the present invention can implement each process implemented by the underwater VR image processing method in the above method embodiment, and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0119] See also Figure 4 , Figure 4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 4As shown, it includes: a memory 402, a processor 401, and a computer program of an underwater VR image processing method stored in the memory 402 and executable on the processor 3401, wherein:

[0120] The processor 401 is used to call the computer program stored in the memory 402 to perform the following steps:

[0121] Acquiring the underwater environment information, the first underwater image, and the second underwater image;

[0122] generating a third underwater image according to the underwater environment information, the first underwater image, and the second underwater image;

[0123] A target VR image is generated based on the third underwater image and a preset three-dimensional underwater space. The three-dimensional underwater space and the area where the VR device is located have the same activity space. The target VR image is used to be displayed in the image display unit.

[0124] Optionally, the underwater environment sensing unit includes at least one of an underwater depth sensor, an underwater pressure sensor, an underwater light sensor, an underwater temperature sensor, and a water transparency detector, the underwater environment information includes at least one underwater environment information such as underwater depth information, underwater pressure information, underwater light information, underwater temperature information, and water transparency information, and the processor 401 generates a third underwater image according to the underwater environment information, the first underwater image, and the second underwater image, including:

[0125] encoding and quantizing the underwater environment information using a pixel value range of the first underwater image or the second underwater image to obtain a noise map;

[0126] The noise map, the first underwater image and the second underwater image are processed by a preset image generation network to generate the third underwater image.

[0127] Optionally, the preset image generation network includes a first image generation network, a second image generation network, and a third image generation network, and the processor 401 performs image generation processing on the noise map, the first underwater image, and the second underwater image through the preset generation network to generate the third underwater image, including:

[0128] Performing image generation processing on the noise map and the first underwater image through the first image generation network to obtain a first generated image, and performing image generation processing on the noise map and the second underwater image through the second image generation network to obtain a second generated image;

[0129] The first generated image and the second generated image are processed by the third image generation network to obtain a third underwater image.

[0130] Optionally, the processor 401 generates a target VR image according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space, including:

[0131] adjusting the preset three-dimensional underwater space according to the third underwater image to obtain an adjusted three-dimensional underwater space;

[0132] calculating virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space according to the third underwater image;

[0133] A target VR image is generated according to the adjusted virtual shooting parameters of the three-dimensional underwater space.

[0134] Optionally, the underwater VR device includes a master device and a slave device, the master device includes a first waterproof shell, a processor unit, an image display unit, an underwater environment sensing unit, and a first image acquisition unit arranged in the first waterproof shell, the slave device includes a second waterproof shell, and a second image acquisition unit arranged in the second waterproof shell, the master device is worn on a first specific part of the human body, the slave device is worn on a second specific part of the human body, the second image acquisition unit is connected to the processor unit by signal, and the processor 401 performs the calculation of the virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space according to the third underwater image, including:

[0135] Determining a user visual parameter of the underwater VR device in a current state according to a size parameter of the third underwater image and an intersection-and-union ratio of the first underwater image and the second underwater image;

[0136] The virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space are calculated according to the user visual parameters.

[0137] Optionally, the determining, by the processor 401, of a user visual parameter of the underwater VR device in a current state according to a size parameter of the third underwater image and an intersection-and-union ratio of the first underwater image and the second underwater image includes:

[0138] Calculating a user posture of the underwater VR device in a current state according to a size parameter of the third underwater image and an intersection-and-union ratio of the first underwater image and the second underwater image;

[0139] According to the user posture, the user visual parameters of the underwater VR device in the current state are determined.

[0140] Optionally, after the step of generating a target VR image according to the third underwater image and a preset three-dimensional underwater space, the method executed by the processor 401 includes:

[0141] Calculate the color difference pixels appearing in the target VR image;

[0142] Analyze the color difference of the color difference pixel point;

[0143] Performing color difference compensation on the color difference pixel points with the opposite color of the color difference to obtain an image to be displayed;

[0144] The image to be displayed is transmitted to the image display unit for display.

[0145] It should be noted that the electronic device provided by the embodiment of the present invention can be applied to VR devices, smart phones, computers, servers and other devices that can perform underwater VR image processing.

[0146] The electronic device provided in the embodiment of the present invention can implement each process implemented by the underwater VR image processing method in the above method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0147] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the underwater VR image processing method or the application-side underwater VR image processing method provided in the embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0149] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. An underwater VR image processing method, characterized in that: The invention is applied to an underwater VR device, wherein the underwater VR device comprises a waterproof housing, a processor unit, an image display unit, an underwater environment sensing unit, a first image acquisition unit and a second image acquisition unit arranged in the waterproof housing, wherein the underwater environment sensing unit is used to acquire underwater environment information of an area where the underwater VR device is located, the first image acquisition unit is used to acquire a first underwater image of a first area, the second image acquisition unit is used to acquire a second underwater image of a second area, the intersection-and-union ratio of the first area and the second area is less than a preset value, the underwater environment sensing unit comprises at least one of an underwater depth sensor, an underwater pressure sensor, an underwater light sensor, an underwater temperature sensor and a water quality transparency detector, the underwater environment information comprises at least one of underwater depth information, underwater pressure information, underwater light information, underwater temperature information and water quality transparency information, and the method comprises the following steps: Acquiring the underwater environment information, the first underwater image, and the second underwater image; encoding and quantizing the underwater environment information using a pixel value range of the first underwater image or the second underwater image to obtain a noise map; Performing image generation processing on the noise image, the first underwater image, and the second underwater image through a preset image generation network to generate a third underwater image; A target VR image is generated based on the third underwater image and a preset three-dimensional underwater space. The three-dimensional underwater space and the area where the VR device is located have the same activity space. The target VR image is used to be displayed in the image display unit.

2. The method according to claim 1, characterized in that The preset image generation network includes a first image generation network, a second image generation network, and a third image generation network. The image generation processing is performed on the noise map, the first underwater image, and the second underwater image by the preset image generation network to generate the third underwater image, including: Performing image generation processing on the noise map and the first underwater image through the first image generation network to obtain a first generated image, and performing image generation processing on the noise map and the second underwater image through the second image generation network to obtain a second generated image; The first generated image and the second generated image are processed by the third image generation network to obtain a third underwater image.

3. The method according to claim 1, characterized in that The method of generating a target VR image according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space, includes: adjusting the preset three-dimensional underwater space according to the third underwater image to obtain an adjusted three-dimensional underwater space; calculating virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space according to the third underwater image; A target VR image is generated according to the adjusted virtual shooting parameters of the three-dimensional underwater space.

4. The method according to claim 3, characterized in that The underwater VR device includes a master device and a slave device, wherein the master device includes a first waterproof housing, a processor unit, an image display unit, an underwater environment sensing unit, and a first image acquisition unit arranged in the first waterproof housing, and the slave device includes a second waterproof housing, and a second image acquisition unit arranged in the second waterproof housing. The master device is worn on a first specific part of a human body, and the slave device is worn on a second specific part of a human body. The second image acquisition unit is connected to the processor unit by signal. The virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space are calculated according to the third underwater image, including: Determining a user visual parameter of the underwater VR device in a current state according to a size parameter of the third underwater image and an intersection-and-union ratio of the first underwater image and the second underwater image; The virtual shooting parameters of the virtual camera in the adjusted three-dimensional underwater space are calculated according to the user visual parameters.

5. The method according to claim 4, characterized in that The determining, according to the size parameter of the third underwater image and the intersection-and-union ratio of the first underwater image and the second underwater image, the user visual parameter of the underwater VR device in the current state comprises: Calculating a user posture of the underwater VR device in a current state according to a size parameter of the third underwater image and an intersection-and-union ratio of the first underwater image and the second underwater image; According to the user posture, the user visual parameters of the underwater VR device in the current state are determined.

6. The method according to claim 1, characterized in that After the step of generating a target VR image according to the third underwater image and the preset three-dimensional underwater space, the method includes: Calculate the color difference pixels appearing in the target VR image; Analyze the color difference of the color difference pixel point; Performing color difference compensation on the color difference pixel points with the opposite color of the color difference to obtain an image to be displayed; The image to be displayed is transmitted to the image display unit for display.

7. An underwater VR image processing device, characterized in that: Applied to an underwater VR device, the underwater VR device comprises a waterproof housing, an underwater VR image processing device, an image display unit, an underwater environment sensing unit, a first image acquisition unit and a second image acquisition unit arranged in the waterproof housing, the underwater environment sensing unit is used to acquire underwater environment information of an area where the underwater VR device is located, the first image acquisition unit is used to acquire a first underwater image of a first area, the second image acquisition unit is used to acquire a second underwater image of a second area, the intersection-and-union ratio of the first area and the second area is less than a preset value, the underwater environment sensing unit comprises at least one of an underwater depth sensor, an underwater pressure sensor, an underwater light sensor, an underwater temperature sensor and a water quality transparency detector, the underwater environment information comprises at least one of underwater depth information, underwater pressure information, underwater light information, underwater temperature information and water quality transparency information, and the device comprises: An acquisition module, used to acquire the underwater environment information, the first underwater image and the second underwater image; A first generating module, configured to encode and quantize the underwater environment information using a pixel value range of the first underwater image or the second underwater image to obtain a noise map; Performing image generation processing on the noise image, the first underwater image, and the second underwater image through a preset image generation network to generate a third underwater image; The second generating module is used to generate a target VR image according to the third underwater image and a preset three-dimensional underwater space, wherein the three-dimensional underwater space and the area where the VR device is located have the same activity space.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the underwater VR image processing method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the underwater VR image processing method according to any one of claims 1 to 6 are implemented.

Images