Virtual environment generation method and related devices

Abstract

The application provides a virtual environment generation method, which comprises the following steps: acquiring a real environment image at a current time and a virtual environment image at a previous time, wherein the real environment image and the virtual environment image are both panoramic images; extracting a target feature of the real environment based on the real environment image at the current time, and extracting a target feature of the virtual environment based on the virtual environment image at the previous time; and generating a virtual environment image at the current time according to the target feature of the real environment and the target feature of the virtual environment. The application generates the virtual environment image at the current time by using the target feature of the real environment and the target feature of the virtual environment, reduces the data volume of image processing, improves the virtual environment generation speed, dynamically generates the virtual environment, and thus improves the real-time performance of the virtual environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the virtual reality field, and particularly relates to a virtual environment generation method and related equipment. BACKGROUND

[0002] With the development of image processing technology and image processing hardware, the virtual reality (VR) technology can be implemented. The VR technology is mainly implemented by using a computer model to generate a graphic image. In a VR system, binocular stereovision plays a great role. Different images seen by two eyes of a user are generated and displayed on different displays. Some systems use a single display, but after a user wears special glasses, only one eye can see odd frame images and the other eye can see even frame images. The difference between the odd frame and the even frame is the parallax, which generates a stereoscopic effect. Since real-time interaction and interaction space of a user need to be considered in the VR system, the image seen by the user has a high real-time requirement. When the direction of the user's gaze changes rapidly, the generation speed of the image cannot keep up, which causes image blurring. SUMMARY

[0003] The present application provides a virtual environment generation method. Target features of a real environment and target features of a virtual environment are obtained by performing feature extraction on a panoramic image of the real environment and a panoramic image of the virtual environment. A virtual environment image at a current time is generated by using the target features of the real environment and the target features of the virtual environment. The data amount of image processing is reduced, the virtual environment generation speed is improved, the virtual environment can be dynamically generated, and the real-time performance of the virtual environment is improved.

[0004] In a first aspect, the present application provides a virtual environment generation method, which comprises:

[0005] obtaining a real environment image at a current time and a virtual environment image at a previous time, wherein the real environment image and the virtual environment image are both panoramic images;

[0006] extracting target features of the real environment based on the real environment image at the current time, and extracting target features of the virtual environment based on the virtual environment image at the previous time;

[0007] generating a virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment.

[0008] Optionally, before the step of obtaining the real environment image at the current time and the virtual environment image at the previous time, the method further comprises:

[0009] obtaining a real environment image at an initial time, and determining a stylization factor of the virtual environment;

[0010] performing feature extraction on the real environment image at the initial time to obtain initial target features;

[0011] performing stylization processing on the initial target features by the stylization factor of the virtual environment to generate a virtual environment image at the initial time.

[0012] Optionally, the steps of extracting target features of the real environment based on the real environment image at the current time, and extracting target features of the virtual environment based on the virtual environment image at the previous time include:

[0013] performing feature extraction on the real environment image at the current time by a preset first feature extraction network to obtain N first target features of the real environment;

[0014] performing feature extraction on the virtual environment image at the previous time by a preset second feature extraction network to obtain M second target features of the virtual environment.

[0015] Optionally, the step of generating a virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment includes:

[0016] based on the stylization factor, matching N first target features with M second target features to obtain n first target features and m second target features that are not successfully matched, wherein n is less than or equal to N, and m is less than or equal to M;

[0017] generating a virtual environment at the current time on the virtual environment image at the previous time according to the n first target features and the m second target features that are not successfully matched.

[0018] Optionally, the step of matching N first target features with M second target features based on the stylization factor to obtain n first target features and m second target features that are not successfully matched includes:

[0019] performing stylization processing on N first target features by the stylization factor to obtain N stylized features corresponding to the first target features;

[0020] iteratively calculating similarities between N stylized features and M second target features, determining the stylized features and the second target features with similarities greater than a preset similarity as successfully matched, and determining the remaining stylized features and the second target features as not successfully matched;

[0021] determine the first target feature corresponding to the stylized feature which is not matched successfully.

[0022] Optionally, the step of generating the virtual environment of the current time on the virtual environment image of the last time according to the n first target features and the m second target features which are not matched successfully comprises:

[0023] determining whether there is a position conflict in the n first target features and the m second target features which are not matched successfully according to the mapping position of each first target feature and the position of each second target feature in the virtual environment image of the last time;

[0024] if there is a position conflict, determining whether the region where the first target feature existing the conflict is completely covered by the region where the second target feature existing the conflict;

[0025] if the first target feature existing the conflict is completely covered, adding the stylized feature corresponding to the first target feature existing the conflict as a foreground feature to the virtual environment image;

[0026] if the first target feature existing the conflict is not completely covered, expanding the region where the first target feature existing the conflict to the size completely covering the region where the second target feature existing the conflict to obtain an expanded region of the first target feature existing the conflict;

[0027] styling the expanded region of the first target feature existing the conflict by the stylization factor to obtain the stylized feature corresponding to the expanded region of the first target feature existing the conflict;

[0028] adding the stylized feature corresponding to the expanded region of the first target feature existing the conflict as a foreground feature to the virtual environment image;

[0029] if there is no position conflict, adding the stylized feature corresponding to the first target feature as a foreground feature to the virtual environment image, and

[0030] mapping the region where the second target feature is to the real environment image of the current time to obtain a mapping region of the second target feature;

[0031] styling the mapping region of the second target feature by the stylization factor to obtain the stylized feature of the mapping region of the second target feature, and adding the stylized feature of the mapping region of the second target feature as a background to the virtual environment image.

[0032] Optionally, the step of generating the virtual environment image of the current moment according to the target feature of the real environment and the target feature of the virtual environment further comprises:

[0033] calculating a displacement amount between the first target feature and the corresponding second target feature that are matched successfully;

[0034] calculating a displacement region between the first target feature and the corresponding second target feature that are matched successfully according to the displacement amount;

[0035] mapping the displacement region into the real environment image of the current moment to obtain a mapping region corresponding to the displacement region;

[0036] styling the mapping region corresponding to the displacement region by the styling factor to obtain a styled feature of the mapping region corresponding to the displacement region, and adding the styled feature of the mapping region corresponding to the displacement region to the virtual environment image as a background.

[0037] In a second aspect, the present application provides a virtual environment generation device, the device comprising:

[0038] an acquisition module configured to acquire a real environment image of a current moment and a virtual environment image of a previous moment, the real environment image and the virtual environment image being panorama images;

[0039] an extraction module configured to extract a target feature of a real environment based on the real environment image of the current moment, and extract a target feature of a virtual environment based on the virtual environment image of the previous moment;

[0040] a generation module configured to generate a virtual environment image of the current moment according to the target feature of the real environment and the target feature of the virtual environment.

[0041] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the virtual environment generation method provided by the present application when executing the computer program.

[0042] In a fourth aspect, a computer readable storage medium is provided, wherein the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the steps of the virtual environment generation method provided by the present application.

[0043] In this invention, a real-world environment image at the current moment and a virtual environment image from the previous moment are acquired, both of which are panoramic images. Based on the real-world environment image at the current moment, target features of the real-world environment are extracted, and based on the virtual environment image from the previous moment, target features of the virtual environment are extracted. The virtual environment image at the current moment is generated based on the target features of the real-world environment and the target features of the virtual environment. By extracting features from the panoramic images of the real-world and virtual environments, target features of the real-world and virtual environments are obtained. Using these target features to generate the virtual environment image at the current moment reduces the amount of data processed by the image processing, increases the speed of virtual environment generation, and allows for dynamic generation of the virtual environment, thereby improving the real-time performance of the virtual environment. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of a virtual environment generation method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the structure of a virtual environment generation device provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figure 1 , Figure 1 This is a flowchart of a virtual environment generation method provided in an embodiment of the present invention, such as... Figure 1 As shown, the virtual environment generation method includes the following steps:

[0050] 101. Obtain the real environment image at the current moment and the virtual environment image at the previous moment.

[0051] In the embodiment of the present application, the virtual environment generation method described above can be used in a VR device, the VR device is provided with a panoramic camera, and the real environment image and the virtual environment image are both panoramic images. The panoramic camera on the VR device is used to take the environment around the user in real time to obtain the real environment image at the current time. In the VR device, the virtual environment image at the last time can be stored, and the virtual environment can be a stylized virtual environment generated according to the environment around the user. The panoramic image of the virtual environment can be obtained by image processing method, such as taking the virtual environment in real time by a virtual camera to obtain the virtual environment image at the last time.

[0052] 102、Based on the real environment image at the current time, the target feature of the real environment is extracted, and based on the virtual environment image at the last time, the target feature of the virtual environment is extracted.

[0053] In the embodiment of the present application, the target feature can be the feature of a foreground object, and the foreground object can be a movable target object, such as a movable human body or object, and the immovable object is a background object.

[0054] Further, the feature extraction network can be used to extract the target feature in the real environment image at the current time, and the target feature in the virtual environment image at the last time.

[0055] It should be noted that the target object in the real environment and the target object in the virtual environment have different displays. For example, the target object in the real environment is a cat, and the target object corresponding to the cat in the virtual environment can be a tiger or a lion. At this time, the target feature in the real environment is the feature of the cat, and the target feature in the virtual environment is the feature of the tiger or the lion. The target object in the real environment is a table, and the target object corresponding to the table in the virtual environment can be a rock or a coral. At this time, the target feature in the real environment is the feature of the table, and the target feature in the virtual environment is the feature of the rock or the coral. This is determined by the stylization of the virtual environment.

[0056] 103、According to the target feature of the real environment and the target feature of the virtual environment, the virtual environment image at the current time is generated.

[0057] In the embodiment of the present application, at the same time, the target features in the real environment and the target features in the virtual environment have a one-to-one correspondence relationship, and at adjacent times, the target features in the real environment and the target features in the virtual environment do not necessarily all correspond, so that the target features in the real environment without a corresponding relationship can be taken as new targets, and the target features in the virtual environment without a corresponding relationship can be taken as removal targets, the new targets are added in the virtual environment image at the last time, and the removal targets are removed, so as to obtain the virtual environment image at the current time.

[0058] In the embodiment of the present application, the real environment image at the current time and the virtual environment image at the last time are obtained, and the real environment image and the virtual environment image are both panoramic images; target features of the real environment are extracted based on the real environment image at the current time, and target features of the virtual environment are extracted based on the virtual environment image at the last time; and a virtual environment image at the current time is generated according to the target features of the real environment and the target features of the virtual environment. By extracting features from the panoramic image of the real environment and the panoramic image of the virtual environment, the target features of the real environment and the target features of the virtual environment are obtained, the target features of the real environment and the target features of the virtual environment are used to generate the virtual environment image at the current time, the data amount of image processing is reduced, the virtual environment generation speed is improved, the virtual environment can be dynamically generated, and the real-time performance of the virtual environment is improved.

[0059] Optionally, before the step of obtaining the real environment image at the current time and the virtual environment image at the last time, the real environment image at the initial time can also be obtained, and the stylization factor of the virtual environment is determined; the initial target features are obtained by performing feature extraction on the real environment image at the initial time; and the initial target features are stylized by the stylization factor of the virtual environment to generate the virtual environment image at the initial time.

[0060] In the embodiment of the present application, when the VR device is started, the environment around the user can be photographed by the panoramic camera on the VR device, so as to obtain the real environment image at the initial time. A plurality of types of styles are provided in the VR device, and the corresponding stylized environment can be generated according to the real environment around the user, so as to obtain the stylized virtual environment. The user can select the required style type by himself.

[0061] Different types of styles correspond to different types of stylization factors, and the real environment image at the initial time can be stylized according to different types of stylization factors to obtain the virtual environment image at the initial time. The processing process can be represented by y=w ● x+b, wherein x is the real environment image at the initial time, w and b are the stylization factors, and y is the virtual environment image at the initial time.

[0062] Further, each type of style corresponds to a set of stylization factors, the set of stylization factors including a plurality of stylization factors, different types of target features corresponding to a stylization factor, and the set of stylization factors further including a background stylization factor. After feature extraction is performed on the real environment at the initial time, a plurality of initial target features are obtained, the types of the initial target features are identified, the corresponding stylization factors are determined according to the types of the initial target features, the initial target features are stylized according to the corresponding stylization factors, and the background in the real environment image is stylized through the background stylization factor, so as to obtain the virtual environment image at the initial time. The background is a remaining area other than the area where the initial target features are located.

[0063] Optionally, in the steps of extracting target features of the real environment based on the real environment image at the current time and extracting target features of the virtual environment based on the virtual environment image at the previous time, the first feature extraction network can be used to perform feature extraction on the real environment image at the current time to obtain N first target features of the real environment; and the second feature extraction network can be used to perform feature extraction on the virtual environment image at the previous time to obtain M second target features of the virtual environment.

[0064] In the embodiments of the present application, the first feature extraction network and the second feature extraction network described above can be feature extraction networks constructed based on a deep convolutional neural network, such as YOLO-V, R-CNNFast, etc. The first feature extraction network described above is trained to extract target features in a real environment image, and the second feature extraction network is trained to extract target features in a virtual environment image. Specifically, the first feature extraction network and the second feature extraction network can be supervised trained by collecting different sample image sets, the sample images used to train the first feature extraction network are images in a real environment, and the sample images used to train the second feature extraction network are images in a virtual environment.

[0065] The real environment image at the current time can be input into the first feature extraction network for feature extraction to obtain first target features in the real environment, and the first target features can be contour features of real target objects. The virtual environment image at the previous time can be input into the second feature extraction network for feature extraction to obtain second target features in the virtual environment, and the second target features can be contour features of virtual target objects.

[0066] In a possible embodiment, the first target feature can be a detection box (x1, y1, w1, h1, u1, r1) of a real target object, where (x1, y1) is a center position of the detection box of the real target object, w1 is a width of the detection box of the real target object, h1 is a height of the detection box of the real target object, u1 is a category of the real target object, and r1 is a confidence of the detection box of the real target object. The second target feature can be a detection box (x2, y2, w2, h2, u2, r2) of a virtual target object, where (x2, y2) is a center position of the detection box of the virtual target object, w2 is a width of the detection box of the virtual target object, h2 is a height of the detection box of the virtual target object, u2 is a category of the virtual target object, and r2 is a confidence of the detection box of the virtual target object.

[0067] Optionally, in the step of generating the virtual environment image at the current moment according to the target feature of the real environment and the target feature of the virtual environment, the N first target features and the M second target features can be matched based on the stylization factor to obtain n first target features and m second target features that fail to be matched, where n is less than or equal to N, and m is less than or equal to M; and the virtual environment at the current moment is generated on the virtual environment image at the previous moment according to the n first target features and the m second target features that fail to be matched.

[0068] In the embodiment of the present application, if there is a first target feature that fails to be matched, it can be determined that the real target object corresponding to the first target feature is a newly added target object, and further, a virtual target object corresponding to the first target feature can be generated on the virtual environment image at the previous moment; if there is a second target feature that fails to be matched, it can be determined that the virtual target object corresponding to the second target feature is a target object that needs to be removed, and further, the virtual target object corresponding to the second target feature can be removed from the virtual environment image at the previous moment.

[0069] Optionally, in the step of matching the N first target features and the M second target features based on the stylization factor to obtain n first target features and m second target features that fail to be matched, the N first target features can be stylized by using the stylization factor to obtain N stylized features corresponding to the first target features; the similarity between the N stylized features and the M second target features is calculated, and the stylized features and the second target features with a similarity greater than a preset similarity are determined to be matched successfully, and the remaining stylized features and the second target features are determined to fail to be matched; and the corresponding first target features are determined according to the stylized features that fail to be matched.

[0070] In the embodiments of the present application, the corresponding stylization factor can be matched according to the type of the first target feature, the corresponding first target feature is stylized by the matched stylization factor, thereby obtaining the stylized feature corresponding to each first target feature, the similarity between the stylized feature corresponding to each first target feature and each second target feature is calculated, thereby obtaining the similarity between the stylized feature corresponding to each first target feature and each second target feature, the greater the similarity, the more similar the stylized feature corresponding to the first target feature and the second target feature, when the similarity is greater than a preset similarity, it can be considered that the stylized feature of the first target feature is the second target feature, and then it can be determined that the first target feature and the corresponding second target feature are matched successfully; when the similarity between the stylized feature of a certain first target feature and any one second target feature is less than the preset similarity, it can be considered that the first target feature is the feature of a new target object, and then it can be determined that the first target feature is not matched successfully, similarly, when the similarity between a certain second target feature and the stylized feature of any one first target feature is less than the preset similarity, it can be considered that the second target feature is the feature of a target object to be removed, and then it can be determined that the second target feature is not matched successfully. The above-mentioned preset similarity can be set by the user, for example, the user can set the preset similarity to 95% or other values, the higher the preset similarity, the higher the matching accuracy.

[0071] Optionally, in the step of generating the virtual environment of the current moment on the virtual environment image of the previous moment according to the n first target features and the m second target features which are not matched successfully, whether there is a position conflict between the n first target features and the m second target features which are not matched successfully can be determined according to the mapping positions of the first target features and the positions of the second target features in the virtual environment image of the previous moment; if there is a position conflict, whether the region where the first target feature exists is completely covered by the region where the corresponding second target feature exists is determined; if it is completely covered, the stylized feature corresponding to the first target feature which exists in conflict is added to the virtual environment image as a foreground feature; if it is not completely covered, the region where the first target feature exists in conflict is expanded to the size of the region where the corresponding second target feature exists in conflict to obtain an expanded region of the first target feature which exists in conflict; the expanded region of the first target feature which exists in conflict is stylized by the stylization factor to obtain the stylized feature corresponding to the expanded region of the first target feature which exists in conflict; the stylized feature corresponding to the expanded region of the first target feature which exists in conflict is added to the virtual environment image as a foreground feature; if there is no position conflict, the stylized feature corresponding to the first target feature is added to the virtual environment image as a foreground feature, and the region where the second target feature exists is mapped to the real environment image of the current moment to obtain a mapping region of the second target feature; the mapping region of the second target feature is stylized by the stylization factor to obtain the stylized feature of the mapping region of the second target feature, and the stylized feature of the mapping region of the second target feature is added to the virtual environment image as a background.

[0072] In the embodiment of the application, the coordinate system of the virtual environment can be set to be the same as the coordinate system of the real environment, and the real environment image and the virtual environment image can be aligned by aligning the coordinate system of the virtual environment with the coordinate system of the real environment.

[0073] When the first target feature is a contour feature, the position of the center point of the contour corresponding to the first target feature in the real environment image of the current moment can be determined as the position of the first target feature, and the position of the first target feature is mapped to the aligned virtual environment image to obtain the mapping position of each first target feature in the virtual environment image of the previous moment. The position of the center point of the contour corresponding to each second target feature in the virtual environment image of the previous moment can be determined.

[0074] If the mapping position of each first target feature in the virtual environment image of the previous time and the position of the second target feature in the virtual environment image of the previous time are obtained, whether there is a position conflict between each first target feature and each second target feature which fails to match successfully. If there is a position conflict between a certain first target feature and a certain second target feature, it indicates that the stylized feature of the first target feature needs to replace the second target feature.

[0075] If there is a position conflict, it is judged whether the region where the first target feature exists completely covers the region where the corresponding second target feature exists. If the first target feature completely covers the second target feature, it indicates that the stylized feature of the first target feature can completely cover the region where the corresponding second target feature exists, and the stylized feature of the first target feature can be directly used as the foreground feature to replace the second target feature. If the first target feature does not completely cover the second target feature, it indicates that the stylized feature of the first target feature cannot completely cover the region where the corresponding second target feature exists, and the region where the first target feature exists needs to be expanded so that the stylized feature of the first target feature can completely cover the region where the corresponding second target feature exists after expansion. The stylized feature of the expanded region of the first target feature is added to the virtual environment image as the foreground feature.

[0076] If there is no position conflict, for the first target feature which fails to match successfully, it can be determined as a newly added target, and the stylized feature of the first target feature can be directly added to the virtual environment image of the previous time as the foreground feature. For the second target feature which fails to match successfully, it can be determined as a target to be removed, and the second target feature can be replaced by the background of the virtual environment. Specifically, the region where the second target feature exists is mapped to the real environment image of the current time to obtain the mapping region of the second target feature; the mapping region of the second target feature is stylized by the stylization factor to obtain the stylized feature of the mapping region of the second target feature, and the stylized feature of the mapping region of the second target feature is added to the virtual environment image as the background.

[0077] Optionally, in the step of generating the virtual environment image of the current time according to the target features of the real environment and the target features of the virtual environment, the displacement amount between the first target feature which matches successfully and the corresponding second target feature can be calculated; the displacement region between the first target feature which matches successfully and the corresponding second target feature is calculated according to the displacement amount; the displacement region is mapped to the real environment image of the current time to obtain the mapping region corresponding to the displacement region; the mapping region corresponding to the displacement region is stylized by the stylization factor to obtain the stylized feature of the mapping region corresponding to the displacement region, and the stylized feature of the mapping region corresponding to the displacement region is added to the virtual environment image as the background.

[0078] In the embodiment of the present application, for the matched first target feature and the corresponding second target feature, since the first target feature can be a human body or a movable object, if there is a displacement between the second target feature at the previous time and the first target feature at the current time, it indicates that the human body or the object moves and interacts with the space during the movement, which will change the background in the displacement interval. Therefore, the displacement between the matched first target feature and the corresponding second target feature can be calculated, the displacement region between the matched first target feature and the corresponding second target feature can be calculated according to the displacement, the mapping region corresponding to the displacement region can be obtained by mapping the displacement region to the real environment image at the current time, the mapping region corresponding to the displacement region can be stylized by the background stylization factor, the stylized feature of the mapping region corresponding to the displacement region can be obtained, and the stylized feature of the mapping region corresponding to the displacement region can be added to the virtual environment image as the background. In this way, the background of the virtual environment can be dynamically generated, so that the generated background in the virtual environment is more accurate.

[0079] Optionally, please refer to Figure 2 , Figure 2 is a structure diagram of a virtual environment generation device provided by the embodiment of the present application, as shown in Figure 2 , the device comprises:

[0080] The acquisition module 201 is configured to acquire a real environment image at a current time and a virtual environment image at a previous time, wherein the real environment image and the virtual environment image are both panoramic images.

[0081] The extraction module 202 is configured to extract a target feature of a real environment based on the real environment image at the current time, and extract a target feature of a virtual environment based on the virtual environment image at the previous time.

[0082] The generation module 203 is configured to generate a virtual environment image at the current time according to the target feature of the real environment and the target feature of the virtual environment.

[0083] Optionally, the device further comprises:

[0084] The initialization acquisition module is configured to acquire a real environment image at an initial time, and determine a stylization factor of the virtual environment.

[0085] The initialization extraction module is configured to perform feature extraction on the real environment image at the initial time to obtain an initial target feature.

[0086] The initialization generation module is configured to perform stylization processing on the initial target feature by the stylization factor of the virtual environment to generate a virtual environment image at the initial time.

[0087] Optionally, the extraction module is further configured to perform feature extraction on the real environment image at the current time by using a preset first feature extraction network to obtain N first target features of the real environment; and perform feature extraction on the virtual environment image at the previous time by using a preset second feature extraction network to obtain M second target features of the virtual environment.

[0088] Optionally, the generation module 202 is further configured to match the N first target features and the M second target features based on the stylization factor to obtain n first target features and m second target features that are not successfully matched, wherein n is less than or equal to N, and m is less than or equal to M; and generate a virtual environment at the current time on the virtual environment image at the previous time according to the n first target features and the m second target features that are not successfully matched.

[0089] Optionally, the generation module 202 is further configured to perform stylization processing on the N first target features by using the stylization factor to obtain N stylized features corresponding to the first target features; iteratively calculate similarities between the N stylized features and the M second target features, determine the stylized features and the second target features with similarities greater than a preset similarity as successfully matched, and determine the remaining stylized features and the second target features as not successfully matched; and determine the corresponding first target features according to the stylized features that are not successfully matched.

[0090] Optionally, the generating module 202 is further configured to determine whether there is a position conflict between the n first target features and the m second target features that have not been matched successfully according to the positions of the respective first target features and the mapping positions of the respective second target features in the virtual environment image at the previous time; if there is a position conflict, determine whether the region where the first target feature that has the conflict is located completely covers the region where the second target feature that has the conflict is located; if the first target feature that has the conflict is completely covered, add the stylized feature corresponding to the first target feature that has the conflict to the virtual environment image as a foreground feature; if the first target feature that has the conflict is not completely covered, expand the region where the first target feature that has the conflict is located to a size that completely covers the region where the second target feature that has the conflict is located to obtain an expanded region of the first target feature that has the conflict; perform stylization processing on the expanded region of the first target feature that has the conflict by using the stylization factor to obtain a stylized feature corresponding to the expanded region of the first target feature that has the conflict; and add the stylized feature corresponding to the expanded region of the first target feature that has the conflict to the virtual environment image as a foreground feature; if there is no position conflict, add the stylized feature corresponding to the first target feature to the virtual environment image as a foreground feature, and map the region where the second target feature is located to the real environment image at the current time to obtain a mapping region of the second target feature; perform stylization processing on the mapping region of the second target feature by using the stylization factor to obtain a stylized feature of the mapping region of the second target feature, and add the stylized feature of the mapping region of the second target feature to the virtual environment image as a background.

[0091] Optionally, the generating module 202 is further configured to calculate a displacement amount between the first target feature that has been matched successfully and the corresponding second target feature; calculate a displacement region between the first target feature that has been matched successfully and the corresponding second target feature according to the displacement amount; map the displacement region to the real environment image at the current time to obtain a mapping region corresponding to the displacement region; perform stylization processing on the mapping region corresponding to the displacement region by using the stylization factor to obtain a stylized feature of the mapping region corresponding to the displacement region, and add the stylized feature of the mapping region corresponding to the displacement region to the virtual environment image as a background.

[0092] The virtual environment generation apparatus provided by the embodiment of the application can implement each process of the virtual environment generation method implemented in the method embodiment, and can achieve the same beneficial effects. To avoid repetition, details are not described herein.

[0093] Referring to Figure 3 , Figure 3Fig. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application, as shown in the figure, comprising: a memory 302, a processor 301, and a computer program of a virtual environment generation method stored on the memory 302 and executable on the processor 301, wherein: Figure 3

[0094] The processor 301 is configured to invoke the computer program stored in the memory 302 to perform the following steps:

[0095] Obtain a real environment image at a current time and a virtual environment image at a previous time, wherein the real environment image and the virtual environment image are both panoramic images;

[0096] Extract target features of the real environment based on the real environment image at the current time, and extract target features of the virtual environment based on the virtual environment image at the previous time;

[0097] Generate a virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment.

[0098] Optionally, before the steps of obtaining the real environment image at the current time and the virtual environment image at the previous time, the method executed by the processor 301 further comprises:

[0099] Obtain a real environment image at an initial time, and determine a stylization factor of the virtual environment;

[0100] Perform feature extraction on the real environment image at the initial time to obtain initial target features;

[0101] Perform stylization processing on the initial target features by using the stylization factor of the virtual environment to generate a virtual environment image at the initial time.

[0102] Optionally, the steps executed by the processor 301 of extracting target features of the real environment based on the real environment image at the current time and extracting target features of the virtual environment based on the virtual environment image at the previous time comprise:

[0103] Perform feature extraction on the real environment image at the current time by using a preset first feature extraction network to obtain N first target features of the real environment;

[0104] Perform feature extraction on the virtual environment image at the previous time by using a preset second feature extraction network to obtain M second target features of the virtual environment.

[0105] Optionally, the step executed by the processor 301 of generating a virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment comprises:​

[0106] match the N first target features with the M second target features based on the stylization factor, to obtain n first target features and m second target features that are not matched successfully, wherein n is less than or equal to N, and m is less than or equal to M;

[0107] generate a virtual environment at a current time on the virtual environment image at the last time according to the n first target features and the m second target features that are not matched successfully.

[0108] Optionally, the step of matching the N first target features with the M second target features based on the stylization factor, to obtain n first target features and m second target features that are not matched successfully, performed by the processor 301, comprises:

[0109] stylizing the N first target features by the stylization factor to obtain N stylized features corresponding to the first target features;

[0110] calculating the similarity between the N stylized features and the M second target features, determining the stylized features and the second target features with a similarity greater than a preset similarity as matched successfully, and determining the remaining stylized features and the second target features as not matched successfully;

[0111] determining the corresponding first target features according to the stylized features that are not matched successfully.

[0112] Optionally, the step of generating a virtual environment at a current time on the virtual environment image at the last time according to the n first target features and the m second target features that are not matched successfully, performed by the processor 301, comprises:

[0113] judging whether there is a position conflict in the n first target features and the m second target features that are not matched successfully according to the position of each first target feature and the mapping position of each second target feature in the virtual environment image at the last time;

[0114] if there is a position conflict, judging whether the region where the first target feature that has the conflict is located completely covers the region where the second target feature that has the conflict is located;

[0115] if the region where the first target feature that has the conflict is located completely covers the region where the second target feature that has the conflict is located, adding the stylized feature corresponding to the first target feature that has the conflict to the virtual environment image as a foreground feature;

[0116] If not completely covered, the region where the first target feature exists conflicts is expanded to the size of the region where the second target feature exists conflicts to obtain an expanded region of the first target feature existing conflicts;

[0117] The expanded region of the first target feature existing conflicts is stylized by the stylization factor to obtain a stylized feature corresponding to the expanded region of the first target feature existing conflicts;

[0118] The stylized feature corresponding to the expanded region of the first target feature existing conflicts is added to the virtual environment image as a foreground feature;

[0119] If there is no position conflict, the stylized feature corresponding to the first target feature is added to the virtual environment image as a foreground feature, and

[0120] The region where the second target feature exists is mapped to a real environment image at the current time to obtain a mapping region of the second target feature;

[0121] The mapping region of the second target feature is stylized by the stylization factor to obtain a stylized feature of the mapping region of the second target feature, and the stylized feature of the mapping region of the second target feature is added to the virtual environment image as a background.

[0122] Optionally, the step of generating the virtual environment image at the current time according to the target feature of the real environment and the target feature of the virtual environment performed by the processor 301 further includes:

[0123] The displacement amount between the first target feature and the corresponding second target feature that match successfully is calculated;

[0124] According to the displacement amount, the displacement region between the first target feature and the corresponding second target feature that match successfully is calculated;

[0125] The displacement region is mapped to a real environment image at the current time to obtain a mapping region corresponding to the displacement region;

[0126] The mapping region corresponding to the displacement region is stylized by the stylization factor to obtain a stylized feature of the mapping region corresponding to the displacement region, and the stylized feature of the mapping region corresponding to the displacement region is added to the virtual environment image as a background.

[0127] The electronic device provided by the embodiment of the present application can realize each process of the virtual environment generation method implemented in the method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0128] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement each process of the virtual environment generation method provided by the embodiments of the present application, and can achieve the same technical effects. To avoid repetition, details are not described here.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments of the methods can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM), etc.

[0130] The above only describes the preferred embodiments of the present application, and of course cannot limit the scope of the right of the present application, so the equivalent changes made according to the claims of the present application still fall within the scope of the present application.

Claims

1. A method of generating a virtual environment, characterized by, The method comprises the following steps: obtaining a real environment image at a current time and a virtual environment image at a previous time, wherein the real environment image and the virtual environment image are both panoramic images; extracting target features of the real environment based on the real environment image at the current time, and extracting target features of the virtual environment based on the virtual environment image at the previous time; generating a virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment; the step of extracting target features of the real environment based on the real environment image at the current time, and extracting target features of the virtual environment based on the virtual environment image at the previous time comprises: extracting features of the real environment image at the current time through a preset first feature extraction network to obtain N first target features of the real environment; extracting features of the virtual environment image at the previous time through a preset second feature extraction network to obtain M second target features of the virtual environment; before the step of obtaining a real environment image at a current time and a virtual environment image at a previous time, the method further comprises: obtaining a real environment image at an initial time, and determining a stylization factor of the virtual environment; extracting features of the real environment image at the initial time to obtain initial target features; styling the initial target features through the stylization factor of the virtual environment to generate a virtual environment image at the initial time; the step of generating a virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment comprises: based on the stylization factor, matching N first target features with M second target features to obtain n first target features and m second target features that are not successfully matched, wherein n is less than or equal to N, and m is less than or equal to M; generating a virtual environment at the current time on the virtual environment image at the previous time according to the n first target features and the m second target features that are not successfully matched.

2. The virtual environment generation method of claim 1, wherein, the step of matching N first target features with M second target features based on the stylization factor to obtain n first target features and m second target features that are not successfully matched comprises: styling N first target features through the stylization factor to obtain N stylized features corresponding to the first target features; iteratively calculating the similarity between N stylized features and M second target features, determining the stylized features and the second target features with a similarity greater than a preset similarity as successfully matched, and determining the remaining stylized features and the second target features as not successfully matched; determining the corresponding first target features according to the stylized features that are not successfully matched.

3. The virtual environment generation method of claim 2, wherein, the step of generating a virtual environment at the current time on the virtual environment image at the previous time according to the n first target features and the m second target features that are not successfully matched comprises: determining whether there is a position conflict between the n first target features and the m second target features that have not been matched successfully according to the mapping positions of the respective first target features and the positions of the respective second target features in the virtual environment image at the previous time; if there is a position conflict, determining whether the region where the first target feature that has the conflict is located completely covers the region where the corresponding second target feature that has the conflict is located; if the first target feature that has the conflict is completely covered, adding the stylized feature corresponding to the first target feature that has the conflict to the virtual environment image as a foreground feature; if the first target feature that has the conflict is not completely covered, expanding the region where the first target feature that has the conflict is located to a size that completely covers the region where the corresponding second target feature that has the conflict is located, to obtain an expanded region of the first target feature that has the conflict; styling the expanded region of the first target feature that has the conflict by using the stylization factor, to obtain a stylized feature corresponding to the expanded region of the first target feature that has the conflict; adding the stylized feature corresponding to the expanded region of the first target feature that has the conflict to the virtual environment image as a foreground feature; if there is no position conflict, adding the stylized feature corresponding to the first target feature to the virtual environment image as a foreground feature, and mapping the region where the second target feature is located to the real environment image at the current time, to obtain a mapping region of the second target feature; styling the mapping region of the second target feature by using the stylization factor, to obtain a stylized feature of the mapping region of the second target feature, and adding the stylized feature of the mapping region of the second target feature to the virtual environment image as a background.

4. The virtual environment generation method of claim 3, wherein, The step of generating the virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment further includes: calculating a displacement amount between the first target feature that has been matched successfully and the corresponding second target feature; calculating a displacement region between the first target feature that has been matched successfully and the corresponding second target feature according to the displacement amount; mapping the displacement region to the real environment image at the current time, to obtain a mapping region corresponding to the displacement region; styling the mapping region corresponding to the displacement region by using the stylization factor, to obtain a stylized feature of the mapping region corresponding to the displacement region, and adding the stylized feature of the mapping region corresponding to the displacement region to the virtual environment image as a background.

5. A virtual environment generation apparatus characterized by comprising: The apparatus includes: an acquisition module configured to acquire a real environment image at a current time and a virtual environment image at a previous time, the real environment image and the virtual environment image being both panoramic images; an extraction module configured to extract target features of a real environment based on the real environment image at the current time, and extract target features of a virtual environment based on the virtual environment image at the previous time; a generation module configured to generate a virtual environment image at the current time according to the target features of the real environment and the target features of the virtual environment. The step of extracting target features of the real environment based on the real environment image of the current moment and extracting target features of the virtual environment based on the virtual environment image of the last moment comprises: performing feature extraction on the real environment image of the current moment through a preset first feature extraction network to obtain N first target features of the real environment; performing feature extraction on the virtual environment image of the last moment through a preset second feature extraction network to obtain M second target features of the virtual environment; Before the steps of obtaining the real environment image of the current moment and the virtual environment image of the last moment, the method further comprises: obtaining a real environment image of an initial moment and determining a stylization factor of the virtual environment; performing feature extraction on the real environment image of the initial moment to obtain initial target features; performing stylization processing on the initial target features through the stylization factor of the virtual environment to generate a virtual environment image of the initial moment; The process of generating the virtual environment image of the current moment according to the target features of the real environment and the target features of the virtual environment comprises: based on the stylization factor, matching the N first target features with the M second target features to obtain n first target features and m second target features that are not successfully matched, wherein n is less than or equal to N and m is less than or equal to M; generating the virtual environment of the current moment on the virtual environment image of the last moment according to the n first target features and the m second target features that are not successfully matched.

6. An electronic device, comprising: comprise: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps in the virtual environment generation method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executable on the processor to implement the steps in the virtual environment generation method according to any one of claims 1 to 4.

Images