Color video generation method and apparatus, and electronic device and storage medium

Abstract

The embodiments of the present invention relate to the technical field of image processing. Disclosed are a color video generation method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a video to be processed, wherein the video to be processed comprises at least one depth map; for each depth map, performing noise diffusion on the depth map by means of a diffusion model, so as to obtain a noise map corresponding to the depth map, wherein the noise map is used for indicating semantic information and depth information of the depth map; and inputting the video to be processed into a color video generation model to generate a corresponding color video, and introducing the noise map that has been encoded by a noise encoder to perform global semantic control on a color video generation process, so as to obtain the color video, wherein the color video comprises at least one color map, and each color map corresponds to one depth map in the video to be processed. The present invention solves the problems in the prior art of the cost being relatively high and the color video generation accuracy being not high.

Description

Color video generation method, device, electronic device and storage medium Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a color video generation method, device, electronic equipment and storage medium. Background Art

[0002] According to statistics from the China Disabled Persons' Federation, there are currently at least 5 million blind people in my country, and with the aging population, the number is increasing annually. By 2050, the number of blind people worldwide is expected to reach 703 million. "Guiding the blind" has always been a hot research topic. Traditional guide glasses have seen continuous market development, with their portability and the accuracy of their assistive algorithms improving. However, maintaining portability while also improving battery life remains a challenge.

[0003] Traditional guide glasses mostly use binocular cameras, RGB color cameras and other modules as their main sensors, and locate objects in indoor scenes through RGBD or RGB semantic segmentation. However, since the power consumption of RGB cameras is much greater than that of depth cameras, the battery life of visually impaired assistive glasses can only be improved by upgrading battery materials. This makes the upgrade cost too high and is the core problem hindering the battery life of guide glasses.

[0004] The power consumption of chips used in depth map acquisition devices is significantly lower than that of chips used in color map acquisition devices. However, when generating high-quality color images based on depth maps, due to the diversity of indoor objects, objects of different categories present different forms, such as hand sanitizer and water cups, and clothes and sheets discarded on the bed. These objects will be ambiguous if only the texture filling of the depth map is used. In addition, the strong changes in texture during image conversion lead to low accuracy in color video generation, which in turn affects the accuracy of downstream recognition tasks.

[0005] Therefore, there is an urgent need for a color video generation method that can reduce costs and improve the accuracy of color video generation. Technical issues

[0006] The embodiments of the present invention provide a color video generation method, device, electronic device, and storage medium to solve the problems of high cost and low accuracy of color video generation methods in related technologies. Technical Solutions

[0007] The technical solution is as follows:

[0008] According to one aspect of the present invention, a color video generation method comprises: obtaining a video to be processed, the video to be processed including at least one depth map; for each depth map, performing noise diffusion on the depth map through a diffusion model to obtain a noise map corresponding to the depth map; the noise map is used to indicate semantic information and depth information of the depth map; the video to be processed is input into a color video generation model to generate a corresponding color video, and a noise map encoded by a noise encoder is introduced to perform global semantic control on the color video generation process to obtain a color video; the color video includes at least one color map, each color map corresponding to a depth map in the video to be processed.

[0009] In one embodiment, noise diffusion is performed on the depth map through a diffusion model to obtain a noise map corresponding to the depth map by the following steps: obtaining a first text prompt word set for the depth map; the first text prompt word is used to describe the category and / or position of each object in the depth map; encoding the first text prompt word into a corresponding text feature vector through a text encoder; adding noise to the depth map in the forward process of the diffusion model, and introducing the text feature vector corresponding to the first text prompt word into the noise adding process of the depth map to obtain a noise map corresponding to the depth map.

[0010] In one embodiment, before performing noise diffusion on the depth map through a diffusion model to obtain a noise map corresponding to the depth map, the method is further implemented by the following steps: training the diffusion model based on a first image data set; training the diffusion model based on the first image data set is implemented by the following steps: acquiring the first image data set; the first image data set includes a depth map sequence and a color map sequence corresponding to the depth map sequence; setting a first training prompt word for each depth map in the depth map sequence; the first training prompt word is used to describe the category and / or position of each object in the depth map; encoding the first training prompt word of each depth map through a text encoder to obtain a text feature vector of each depth map; training the initial diffusion model based on the depth map sequence, the color map sequence and the text feature vector of each depth map until the training is completed, storing the parameters of the diffusion model, and obtaining the trained diffusion model.

[0011] In one embodiment, a color video generation model includes a diffusion encoder and a diffusion decoder; the video to be processed is input into the color video generation model to generate a corresponding color video, and a noise map encoded by a noise encoder is introduced to perform global semantic control on the color video generation process, and the color video is obtained by the following steps: based on the video to be processed, the depth map and its corresponding noise map are obtained, and a second text prompt word set for the target scene where the video to be processed is located is obtained; the second text prompt word is used to describe the target scene; the depth map and the noise map encoded by the noise encoder are input into the diffusion encoder for encoding to obtain an input vector corresponding to the depth map; the input vector and the second text prompt word are decoded by the diffusion decoder to obtain an output vector corresponding to the depth map, and the output vector is decoded twice to obtain a color map corresponding to the depth map; the color maps corresponding to each of the depth maps are subjected to video synthesis processing to obtain the color video.

[0012] In one embodiment, the video to be processed is input into a color video generation model to generate a corresponding color video, and a noise map encoded by a noise encoder is introduced to perform global semantic control on the color video generation process. Before obtaining the color video, the method is further implemented by the following steps: obtaining a second image data set to train a machine learning model until the training is completed, storing the parameters of the machine learning model, and obtaining an initial color video generation model, the initial color video generation model having the ability to convert the video to be processed into the color video; obtaining second training prompt words set for several target scenes; the second training prompt words are used to describe the target scenes; the initial color video generation model is trained a second time based on the obtained second text prompt words until the training is completed, storing the parameters of the color video generation model, and obtaining the trained color video generation model.

[0013] In one embodiment, the color images corresponding to each of the depth maps are subjected to video synthesis processing to obtain the color video by the following steps: the color images corresponding to each of the depth maps are sampled at intervals to obtain multiple key frames; the multiple key frames are grouped to obtain multiple key frame sequences; the key frame sequence includes at least one key frame; and at least one key frame in each of the key frame sequences is fused to obtain the color video.

[0014] In one embodiment, the key frames in each key frame sequence are fused separately to obtain the color video by the following steps: for each key frame sequence, the key frames in the key frame sequence are spliced ​​to obtain a multi-frame fused image; the fused image corresponds one-to-one to the key frame sequence; and the cross-attention calculation of the key frames of the multi-frame fused image is performed through the attention mechanism to obtain the color video.

[0015] In one embodiment, after obtaining the color video, the method is further implemented by the following steps: performing a semantic segmentation task based on the color video to obtain a semantic segmentation result; the semantic segmentation result includes one or more of the position of the object in the target scene and the category of the object.

[0016] According to one aspect of the present invention, a color video generation device includes: a video acquisition module to be processed, used to acquire a video to be processed, wherein the video to be processed includes at least one depth map; a noise diffusion module, used to perform noise diffusion on each depth map through a diffusion model to obtain a noise map corresponding to the depth map; the noise map is used to indicate semantic information and depth information of the depth map; and a color video generation module, used to input the video to be processed into a color video generation model to generate a corresponding color video, and introduce the noise map encoded by a noise encoder to perform global semantic control on the color video generation process to obtain a color video.

[0017] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the color video generation method as described above.

[0018] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the color video generation method as described above.

[0019] According to one aspect of the present invention, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the color video generation method as described above. Beneficial effects

[0020] The beneficial effects brought about by the technical solution provided by the present invention are:

[0021] In the above technical solution, a video to be processed is first obtained, which includes at least one depth map. Then, for each depth map, noise diffusion is performed on each depth map through a diffusion model to obtain a noise map corresponding to the depth map, wherein the noise map is used to indicate the semantic information and depth information of the depth map. The video to be processed is input into a color video generation model to generate a corresponding color video, and the noise map encoded by a noise encoder is introduced to perform global semantic control on the color video generation process. Then, the semantic information and depth information in the depth map can be fused into the color video generation process through the noise map, so that the overall semantic information is added to the color video generation process, and the accuracy of the color video generation process can be improved. Finally, a high-quality color video is obtained, and a high-quality color video is obtained from a depth video with low acquisition cost. Therefore, the present invention can not only ensure the accuracy of color video generation, but also reduce the high cost problem brought about by directly acquiring color video, thereby effectively solving the problems of high cost and low accuracy of color video generation in related technologies.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0024] FIG1 is a schematic diagram showing the result of a color video generation method in the prior art;

[0025] FIG2 is a schematic diagram of an implementation environment according to the present invention;

[0026] FIG3 is a flow chart showing a method for generating a color video according to an exemplary embodiment;

[0027] FIG4 a is a flow chart of step 230 in the embodiment corresponding to FIG3 in one embodiment;

[0028] FIG4 b is a schematic diagram of noise diffusion in an embodiment corresponding to the embodiment of FIG3 ;

[0029] FIG5 is a flow chart of a diffusion model training process in an embodiment corresponding to FIG3 ;

[0030] FIG6 is a flowchart of a method for generating a color video according to another exemplary embodiment;

[0031] FIG7 is a flow chart of step 250 in one embodiment of the embodiment corresponding to FIG3 ;

[0032] FIG8 is a flow chart of step 570 in one embodiment of the embodiment corresponding to FIG7 ;

[0033] FIG9 is a schematic diagram of a color video generation method according to an application scenario;

[0034] FIG10 is a flow chart of a method for generating a color video according to an application scenario;

[0035] FIG11 is a block diagram showing a color video generating apparatus according to an exemplary embodiment;

[0036] FIG12 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0037] Fig. 13 is a block diagram of an electronic device according to an exemplary embodiment.

[0038] The above-mentioned drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present invention in any way, but rather to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. Modes for Carrying Out the Invention

[0039] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0040] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0041] Thanks to the rise of diffusion-based image generation algorithms and the introduction of large pre-trained models, depth images can generate high-quality color images. This makes it possible to obtain images with the same properties as those obtained with RGBD cameras using only a depth camera. At the same time, recognition algorithms for assistive glasses for the visually impaired currently face the following challenges: First, indoor scenes, which predominate in the lives of visually impaired people, often feature objects occluding each other, making objects with similar textures difficult to distinguish. Relying solely on the 2D photometric properties of RGB images cannot accurately separate these objects. While the 3D distance representation of depth images can provide clearly structured spatial information, recognition algorithms based on depth images lack robustness due to the uncertainty and insufficient information captured by depth images.

[0042] Although image generation methods can be used to achieve the same effect as the current RGBD visually impaired assistance algorithm using only depth maps, the depth map only provides structural information, resulting in some objects with similar structures and appearances being filled with objects of other categories. As shown in Figure 1, the first row is the depth map, the second row is the color map corresponding to each depth map in the first row, and the third row is the color map generated by the prior art based on the depth map in the first row. Obviously, for common objects such as tables, stools, beds and other common objects with little change in structural information, it is easy to fill the depth map with texture. However, for objects with different structures such as shampoo and lockers, it is very easy to mistakenly fill them with other items when directly filling them with texture. Such semantic information cannot be provided by the depth map alone, and the texture filling between each frame is uncertain. The rapidly changing texture plus the wrong semantic filling poses a very big challenge to the subsequent recognition algorithm.

[0043] As can be seen from the above, the related technologies still have the defects of high cost and low accuracy and stability in color video generation.

[0044] To this end, the present invention provides a color video generation method that is robust to diverse environments, ensures accurate color video generation, and avoids the cost associated with acquiring color images. This color video generation method is suitable for use in assistive devices for the visually impaired, which can be implemented by integrating an image acquisition device with a color video generation device. The color video generation device can be, for example, a color video generator, and the image acquisition device can be, for example, a camera. The color video generation method in embodiments of the present invention can be applied to a variety of scenarios, such as the daily lives of visually impaired people.

[0045] Figure 2 is a schematic diagram of an implementation environment involved in a color video generation method. It should be noted that this implementation environment is only an example adapted to the present invention and should not be considered as providing any limitation on the scope of application of the present invention.

[0046] The implementation environment includes a collection end 110 and a service end 130 .

[0047] Specifically, the acquisition end 110 can be considered an image acquisition device, including but not limited to electronic devices with shooting capabilities such as webcams, cameras, and camcorders. It can also be considered a device that integrates image acquisition and processing, including but not limited to electronic devices with both shooting and color video generation capabilities, such as laptop computers equipped with webcams. The server end 130 can also be considered a color video generation device, including but not limited to electronic devices with color video generation capabilities such as desktop computers, laptop computers, and servers. It can also be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. The server end 130 is used to provide background services, such as, but not limited to, color video generation services.

[0048] A network communication connection is pre-established between the server 130 and the acquisition terminal 110 via a wired or wireless method, and data transmission between the server 130 and the acquisition terminal 110 is achieved via the network communication connection. The transmitted data includes but is not limited to: videos to be processed, etc.

[0049] In one application scenario, the acquisition end 110 has both image acquisition functions and color video generation functions. After the acquisition end 110 acquires the video to be processed, the color video generation service is called. For each depth map in the video to be processed, noise diffusion is performed on each depth map through a diffusion model to obtain a noise map corresponding to the depth map. The video to be processed is input into the color video generation model to generate the corresponding color video, and the noise map encoded by the noise encoder is introduced to perform global semantic control on the color video generation process to obtain a color video, thereby solving the problem of low accuracy of color video generation in related technologies.

[0050] Of course, in other application scenarios, color video generation can also be achieved through the interaction between the acquisition end 110 and the server end 130.

[0051] In another application scenario, through the interaction between the acquisition terminal 110 and the server 130 , the acquisition terminal 110 acquires a video to be processed and uploads the video to be processed to the server 130 to request the server 130 to provide a color video generation service.

[0052] At this time, for the server 130, after receiving the image target uploaded by the acquisition end 110, it calls the color video generation service, and for each depth map in the video to be processed, performs noise diffusion on each depth map through the diffusion model to obtain a noise map corresponding to the depth map, and inputs the video to be processed into the color video generation model to generate the corresponding color video, and introduces the noise map encoded by the noise encoder to perform global semantic control on the color video generation process to obtain a color video, thereby improving the accuracy of color video generation, thereby solving the problem of low accuracy of color video generation existing in related technologies.

[0053] Please refer to Figure 3. An embodiment of the present invention provides a color video generation method, which is applicable to electronic devices. For example, the electronic device can be an electronic device with both shooting functions and color video generation functions, such as a laptop computer equipped with a camera, or an electronic device with color video generation functions, such as a desktop computer, a laptop computer, a server, etc.

[0054] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0055] As shown in FIG3 , the method may include the following steps:

[0056] Step 210: Obtain the video to be processed.

[0057] The video to be processed includes at least one depth map.

[0058] In one possible implementation, the video to be processed is obtained by capturing and photographing the current environment using an image acquisition device, which may be an electronic device with image acquisition capabilities, such as visually impaired assistive glasses equipped with a camera, a depth camera, and the like.

[0059] It is understood that shooting can be a single shot or continuous shooting. In this case, continuous shooting can produce a video, and the image can be any frame in the video. In the case of multiple shots, multiple photos can be produced, and the image can be any one of the multiple photos. In other words, the image in this embodiment can come from a dynamic image, such as multiple frames in a video or multiple photos, or from a static image, such as any frame in a video or any one of multiple photos. Accordingly, the color video generation in this embodiment can be performed on a frame basis.

[0060] In step 230 , for each depth map, noise diffusion is performed on the depth map using a diffusion model to obtain a noise map corresponding to the depth map.

[0061] In one possible implementation, as shown in FIG4 a , step 230 may include the following steps:

[0062] Step 310: Obtain a first text prompt word set for the depth map.

[0063] The first text prompt word is used to describe the category and / or position of each object in the depth map.

[0064] Step 330: Encode the first text prompt word into a corresponding text feature vector through a text encoder.

[0065] Step 350 , adding noise to the depth map in the forward process of the diffusion model, and introducing the text feature vector corresponding to the first text prompt word into the noise adding process of the depth map to obtain a noise map corresponding to the depth map.

[0066] As shown in Figure 4b, X(n)0 refers to the depth map without noise. r In the process, the depth map is gradually added with noise through the forward process of the diffusion model, and finally the noise map X(n) is obtained. r In Figure 4b, whether X(n)0 or X(n) r In the noise adding process between the two, the first text prompt word encoded by the text encoder is introduced.

[0067] In one possible implementation, the diffusion model is a trained machine learning model with noise diffusion capability. Specifically, the diffusion model is trained using the first image dataset.

[0068] Specifically, as shown in FIG5 , the training process of the diffusion model may include the following steps:

[0069] Step 410: Acquire a first image data set.

[0070] The first image data set includes a depth map sequence and a color map sequence corresponding to the depth map sequence. The depth map sequence includes multiple depth maps, and the color map sequence includes multiple color maps.

[0071] In one possible implementation, the first image dataset is obtained by capturing image data in different target scenes using an image acquisition device. The image acquisition device can be an electronic device with image acquisition capabilities, such as visually impaired assistive glasses equipped with a camera, a depth camera, and the like. The different target scenes can include living environments such as bedrooms, streets, shopping malls, and parks, as well as indoor scenes where visually impaired people live in significant numbers, all of which are not limited here. This ensures that the target scenes in which the image data in the first image dataset reside are sufficiently diverse, thereby improving the robustness of the diffusion model.

[0072] In addition, by training the diffusion model with the first image data set collected in different target scenes, the mapping relationship between the depth map in the depth map sequence and the color map in the color map sequence under various target scenes can be obtained, which is beneficial to improving the accuracy and stability of color video generation.

[0073] Step 430: Set a first training prompt word for each depth map in the depth map sequence.

[0074] Among them, the first training prompt word is used to describe the category and / or position of each object in the depth map. For example, the picture shows a desk between two sofas, and next to the desk is a bookshelf filled with books, etc. Then, the description sentence as the first training prompt word not only involves the category of the object, such as desk, sofa, bookshelf, etc., but also involves the position of the object, such as the desk between two sofas, etc. It is worth mentioning that the first training prompt word can be set manually or automatically identified by calling the machine learning model, and it is not limited here.

[0075] Step 450 : Encode the first training prompt word of each depth map through a text encoder to obtain a text feature vector of each depth map.

[0076] In one possible implementation, the text encoder may include a clip text embedder, TF-IDF, Word2Vec, GloVe, FastText, ELMO, CoVe, BERT, RoBERTa, and other text encoders, which are not limited here.

[0077] Step 470 : Train an initial diffusion model based on the depth map sequence, the color map sequence, and the text feature vectors of each depth map until the training is completed, store the parameters of the diffusion model, and obtain a trained diffusion model.

[0078] In this way, GPU memory is saved by storing the parameters of the diffusion model, further improving training efficiency.

[0079] In one possible implementation, step 470 may include the following steps:

[0080] Step S1: Arrange a depth image sequence and a corresponding color image sequence into a plurality of image pairs according to a corresponding relationship.

[0081] The image pair includes a depth map and a color map corresponding to the depth map.

[0082] In step S2, the color image in each image pair is used as the target image and the depth image is used as the original image. The diffusion model is trained multiple times by adding random noise and the text feature vector corresponding to each depth image to obtain a trained diffusion model.

[0083] In one possible implementation, a noise map sequence corresponding to the depth map sequence is obtained during the training of the diffusion model. An inversion scene name is set for each depth map according to the second text prompt word of each depth map. The inversion scene name corresponds one-to-one to the noise map sequence and the depth map sequence, respectively, and the noise map sequence and the inversion scene name are stored.

[0084] In this embodiment, by using the color image in each image pair as the target image and the depth image as the original image, the color image can be used as a guide to control the training direction of the model, and then the model is further supervised by adding text feature vectors to improve the efficiency and effectiveness of model training. At the same time, random noise is added to avoid overfitting during model training and ensure the robustness of model training.

[0085] In step 250 , the video to be processed is input into a color video generation model to generate a corresponding color video, and a noise map encoded by a noise encoder is introduced to perform global semantic control on the color video generation process to obtain a color video.

[0086] In one possible implementation, the color video generation model includes a diffusion encoder and a diffusion decoder.

[0087] Referring to FIG6 , the process of constructing a color video generation model may include the following steps:

[0088] Step 610: Obtain a second image data set to train a machine learning model until the training is completed, store the parameters of the machine learning model, and obtain an initial color video generation model.

[0089] Step 630: Acquire second training prompt words set for a plurality of target scenes.

[0090] Step 650 : Perform secondary training on the initial color video generation model according to the acquired second text prompt word until the training is completed, store the parameters of the color video generation model, and obtain a trained color video generation model.

[0091] Through the above training process, the first training and second training (i.e., fine-tuning) of the color video generation model are realized, so that the color video generation model has the ability to losslessly convert the processed video into a color video, thereby enabling depth maps of different quality to obtain better color images.

[0092] Based on the trained color video generation model, as shown in FIG7 , step 250 may include the following steps:

[0093] Step 510: Based on the video to be processed, a depth map and its corresponding noise map are obtained, and a second text prompt word set for the target scene where the video to be processed is located is obtained.

[0094] Step 530 : Input the depth map and the noise map encoded by the noise encoder into a diffusion encoder for encoding to obtain an input vector corresponding to the depth map.

[0095] Step 550: Decode the input vector and the second text prompt word through a diffusion decoder to obtain an output vector corresponding to the depth map, and perform secondary decoding on the output vector to obtain a color map corresponding to the depth map.

[0096] Step 570 : Perform video synthesis processing on the color images corresponding to the depth images to obtain a color video.

[0097] In this way, the input vector is directionally controlled in the latent space through the second text prompt word, and the information in the text conditional vector is infiltrated into the input vector to obtain the input vector. The diffusion decoder is used to restore the output vector from the latent space into an image represented by pixels, and then a color image corresponding to the depth map is obtained to complete the color video generation process. The encoder is used to compress the depth map into a low-dimensional input vector in the latent space to facilitate the improvement of calculation speed. At the same time, the inverted scene name is converted into a high-dimensional text conditional vector for easy understanding by the machine learning model, thereby improving the efficiency and accuracy of color video generation.

[0098] Specifically, as shown in FIG8 , step 570 may include the following steps:

[0099] Step 571 : Perform interval sampling on the color image corresponding to each depth image to obtain multiple key frames.

[0100] Step 573: Group the multiple key frames to obtain multiple key frame sequences.

[0101] The key frame sequence includes at least one key frame.

[0102] In one possible implementation, a key frame is set for the color image sequence at intervals of 5 frames to obtain multiple key frames, with each 3 key frames forming a group to obtain multiple key frame sequences. The specific sampling interval can be 5 frames, 6 frames, 7 frames, etc., and the key frame grouping method can be every 3 key frames in a group, every 4 key frames in a group, etc., which are not limited here.

[0103] In this way, the consistency of the filling texture can be controlled, and the generation quality and stability of the color video can be improved.

[0104] Step 575 , fuse the key frames in each key frame sequence to obtain a color video.

[0105] In one possible implementation, step 575 may include the following steps:

[0106] Step S1: For each key frame sequence, the key frames in the key frame sequence are spliced ​​together to obtain a multi-frame fused image.

[0107] Among them, the fused image corresponds to the key frame sequence one by one.

[0108] In step S2, a cross-attention calculation of key frames is performed on the multi-frame fusion image through the attention mechanism to obtain a color video.

[0109] Through the above process, the embodiment of the present invention ensures the control of overall semantics and the stability of video semantics by using a unified attention mechanism between control frames, thereby solving the problems of high cost and low accuracy of color video generation in related technologies.

[0110] Through the above process, the embodiment of the present invention performs noise diffusion on the depth map through a diffusion model to obtain a corresponding noise map, wherein the noise map is used to indicate the semantic information and depth information of the depth map, and then the depth map is encoded into a corresponding input vector through the encoder, and a corresponding text condition vector is obtained based on the noise map, and then the input vector and the text condition vector are calculated by the encoder to obtain an output vector. Then, the semantic information and depth information in the depth map can be integrated into the process of color video generation in the form of a text condition vector through the noise map, so that the overall semantic information is added to the color video generation process, and the accuracy of the color video generation process can be improved. Finally, the output vector is decoded into the corresponding color map through the decoder, and a high-quality color map is obtained from a depth map with low acquisition cost. Therefore, the present invention can not only ensure the accuracy of color video generation, but also reduce the high cost problem brought about by directly acquiring color maps, thereby effectively solving the problems of high cost and low accuracy of color video generation existing in related technologies.

[0111] In an exemplary embodiment, after obtaining the color video, a semantic segmentation task is performed on the color video to obtain a semantic segmentation result, wherein the semantic segmentation result includes one or more of the positions of objects in the target scene and the categories of the objects.

[0112] Specifically, the implementation process of semantic segmentation may include the following steps:

[0113] Step b1: extract features from each frame of the color video to obtain a feature map sequence.

[0114] The feature map sequence includes multiple feature maps, and each feature map corresponds to a video frame in the color video.

[0115] Step b2: for each feature map in the feature map sequence, locate the target in the corresponding video frame according to the feature map, and segment the video frame to obtain a target area map based on the located target position in the video frame.

[0116] Step b3: Based on the target region map, the category of the target in the video frame is predicted to obtain a semantic segmentation result.

[0117] Among them, the semantic segmentation results include the location of the target in the video frame, the target category, etc.

[0118] Through the above process, the embodiment of the present invention can realize the adaptive control of video generation of overall semantic information changes, realize real-time semantic segmentation of indoor scenes by only collecting depth maps when guiding the blind, thereby providing data for the blind, and provide a stable information source of semantic information for downstream recognition tasks such as semantic segmentation by controlling the overall consistency of the color video, thereby reducing the hardware cost of the design of guide glasses and improving the overall design efficiency and accuracy.

[0119] FIG9 is a schematic diagram of a color video generation method in an application scenario. In this application scenario, an electronic device that needs to generate color video may be a depth camera or the like.

[0120] In Figure 9, the parameters of the DDIM inversion model are frozen after training, representing the completed diffusion model. Similarly, the color video generation model, which includes the SD encoder and SD decoder, is obtained by training the machine learning model multiple times. Specifically, after the first training session, the parameters are frozen to form the initial color video generation model. Text prompts are introduced to continue training to fine-tune the frozen parameters. After the second training session, the parameters are frozen to form the completed color video generation model.

[0121] Based on the trained models described above, a text prompt word (i.e., the first text prompt word) is set for each video frame in the input video. The text feature vector corresponding to the first text prompt word is obtained through the text encoder. Each text feature vector is then added to the DDIM inversion module to perform noise diffusion on each video frame, obtaining the noise prompt word (i.e., noise map) corresponding to each video frame.

[0122] When the input video is input to the SD encoder, the noise prompt word is also input to the SD encoder through the noise encoder, and then the input video is encoded into an input vector in the latent space by the SD encoder. Under the action of the text prompt word (i.e., the second text prompt word), the input vector is decoded multiple times in the SD decoder. Combined with the attention mechanism of cross-frame attention, accurate color video can be obtained.

[0123] In one possible implementation, the ControlNet model is used as a color video generation model. In the ControlNet model, the weights of each module are frozen to increase the training speed of each module. By making a trainable copy of the SD encoder, overfitting is avoided when the data set is very small, while maintaining the quality of the large model learned from billions of images. Zero convolution with zero initial weights and bias is used, and a zero convolution is added to each layer in the trainable copy to connect to the corresponding layer of the original network, so that any further optimization will improve the model performance and the training speed is very fast.

[0124] FIG10 is a flow chart of a color video generation method in an application scenario. In this application scenario, electronic devices such as guide glasses may be used for color video generation, and the target scene specifically refers to indoor scenes where a large number of visually impaired people live.

[0125] In Figure 10, facing the real object, the depth map camera module in the guide glasses obtains the depth map sequence to be detected, and text prediction is performed on the indoor scene based on the depth map sequence. Then, based on the text prediction result, a color video is generated through the depth map sequence. Then, semantic segmentation is performed on the color video to obtain the position of each object in the indoor scene and the category information of the object.

[0126] Through the above process, by using the color video generation method provided by the present invention in guide glasses, effective help information is provided to the visually impaired, the hardware cost of the guide glasses design is reduced, and the overall design efficiency and accuracy are improved.

[0127] The following is an embodiment of the device of the present invention, which can be used to execute the color video generation method involved in the present invention. For details not disclosed in the device embodiment of the present invention, please refer to the method embodiment of the color video generation method involved in the present invention.

[0128] Referring to FIG. 11 , an embodiment of the present invention provides a color video generating device 800 .

[0129] The apparatus 800 includes but is not limited to: a to-be-processed video acquisition module 810 , a noise diffusion module 830 , and a color video generation module 850 .

[0130] The to-be-processed video acquisition module 810 is configured to acquire a to-be-processed video, wherein the to-be-processed video includes at least one depth map;

[0131] The noise diffusion module 830 is used to perform noise diffusion on each depth map using a diffusion model to obtain a noise map corresponding to the depth map; the noise map is used to indicate semantic information and depth information of the depth map;

[0132] The color video generation module 850 is used to input the video to be processed into the color video generation model to generate the corresponding color video, and introduce the noise map encoded by the noise encoder to perform global semantic control on the color video generation process to obtain the color video.

[0133] It should be noted that the color video generation provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the color video generation device will be divided into different functional modules to complete all or part of the functions described above.

[0134] In addition, the color video generating device and the color video generating method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0135] FIG12 is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. The electronic device is applicable to the acquisition terminal 110 in the implementation environment shown in FIG2 .

[0136] It should be noted that the electronic device is only an example adapted for the present invention and should not be considered to provide any limitation on the scope of use of the present invention. The electronic device should not be interpreted as needing to rely on or necessarily having one or more components in the exemplary electronic device 2000 shown in FIG. 12 .

[0137] The hardware structure of the electronic device 2000 may vary greatly due to different configurations or performances. As shown in FIG12 , the electronic device 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0138] Specifically, the power supply 210 is used to provide operating voltage for various hardware devices on the electronic device 2000 .

[0139] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices, for example, interacting between the acquisition terminal 110 and the service terminal 130 in the implementation environment shown in FIG2 .

[0140] Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, as shown in FIG12 , which is not specifically limited here.

[0141] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0142] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000 to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0143] Application 253 is a computer-readable instruction that performs at least one specific task based on operating system 251. It may include at least one module (not shown in FIG12 ), each of which may include computer-readable instructions for electronic device 2000. For example, a color video generation device may be considered an application 253 deployed on electronic device 2000.

[0144] The data 255 may be photos, pictures, etc. stored in a disk, or may be diffusion model parameters, etc. stored in the memory 250 .

[0145] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 in the memory 250. For example, the color video generation method is implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0146] In addition, the present invention can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or combination thereof.

[0147] Referring to FIG. 13 , an embodiment of the present invention provides an electronic device 4000 , which may include an image acquisition device having an image acquisition function, etc.

[0148] In FIG. 13 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003 .

[0149] Data exchange between processor 4001 and memory 4003 can be achieved via at least one communication bus 4002. Communication bus 4002 may include a path for transmitting data between processor 4001 and memory 4003. Communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Communication bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG13 shows only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0150] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0151] Processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 4001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0152] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited thereto.

[0153] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0154] The computer-readable instructions are executed by one or more processors 4001 to implement the color video generation method in the above embodiments.

[0155] In addition, an embodiment of the present invention provides a storage medium having computer-readable instructions stored thereon. The computer-readable instructions are executed by one or more processors to implement the color video generation method described above.

[0156] In an embodiment of the present invention, a computer program product is provided. The computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the color video generation method described above.

[0157] Compared with the related art, the present invention has the following beneficial effects:

[0158] 1. The present invention first obtains a video to be processed, which includes at least one depth map. Then, for each depth map, noise diffusion is performed on each depth map through a diffusion model to obtain a noise map corresponding to the depth map, wherein the noise map is used to indicate the semantic information and depth information of the depth map. The video to be processed is input into a color video generation model to generate a corresponding color video, and the noise map encoded by a noise encoder is introduced to perform global semantic control on the color video generation process. Then, the semantic information and depth information in the depth map can be integrated into the color video generation process through the noise map, so that the overall semantic information is added to the color video generation process, and the accuracy of the color video generation process can be improved. Finally, a high-quality color video is obtained, and a high-quality color video is obtained from a depth video with low acquisition cost. Therefore, the present invention can not only ensure the accuracy of color video generation, but also reduce the high cost problem brought about by directly acquiring color video, thereby effectively solving the problems of high cost and low accuracy of color video generation existing in the related art.

[0159] 2. The present invention ensures that the converted texture image changes smoothly, reduces the burden on downstream recognition tasks, and completes the task of generating accurate indoor scene color video and performing recognition using only a depth image group while balancing real-time performance and accuracy.

[0160] 3. The present invention can add control of any scene by modifying a separately trained diffusion inversion module instead of retraining the entire large model, and the method does not require human prompts to generate color video.

[0161] 4. This paper designs a ControlNet based on a large-scale AIGC pre-trained model and adaptive noise prompt matching and control fine-tuning based on diffusion model inversion. Finally, a unified cross-attention module is added between control frames to ensure overall semantic control and video semantic stability. This solves the current problems of high power consumption of color image acquisition chips and changes in the overall RGB semantics that hinder the design of sustainable guide glasses.

[0162] 5. This invention introduces a technology for video generation and semantic structure control of video generation based on a large-scale pre-trained AIGC model that can adaptively control changes in overall semantic information. This technology enables real-time semantic segmentation of indoor scenes by only collecting depth maps when guiding the blind, thereby providing information for guiding the blind.

[0163] 6. This invention provides a new approach to the design of battery-stable guide glasses. It utilizes a lower-power depth map acquisition chip. Through diffusion inversion and manually designed prompt words, a ControlNet fine-tuned for visual noise cue control is trained for a relatively fixed location to generate globally semantically stable color images. This avoids the need for carefully designed prompt words, which hinders the application of ControlNet in assistive glasses for the visually impaired. Furthermore, a keyframe-based cross-frame attention scheme is designed to control the consistency of overall video results, providing a stable source of semantic information for downstream recognition tasks such as segmentation. This achieves the same accuracy as the indoor scene assistive algorithm originally equipped with an RGB(D) acquisition and processing chip. This reduces the hardware cost of guide glasses and improves overall design efficiency.

[0164] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0165] The above descriptions are only partial embodiments of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A color video generation method, characterized in that: The method comprises: Acquire a video to be processed, where the video to be processed includes at least one depth map; For each of the depth maps, noise diffusion is performed on the depth map through a diffusion model to obtain a noise map corresponding to the depth map; the noise map is used to indicate semantic information and depth information of the depth map; The video to be processed is input into a color video generation model to generate a corresponding color video, and a noise map encoded by a noise encoder is introduced to perform global semantic control on the color video generation process to obtain a color video; the color video includes at least one color map, and each color map corresponds to a depth map in the video to be processed.

2. The method according to claim 1, characterized in that The step of performing noise diffusion on the depth map by using a diffusion model to obtain a noise map corresponding to the depth map includes: Acquire a first text prompt word set for the depth map; the first text prompt word is used to describe the category and / or position of each object in the depth map; Encoding the first text prompt word into a corresponding text feature vector by a text encoder; The depth map is denoised in the forward process of the diffusion model, and the text feature vector corresponding to the first text prompt word is introduced into the denoising process of the depth map to obtain a noise map corresponding to the depth map.

3. The method according to claim 1, characterized in that Before performing noise diffusion on the depth map by using a diffusion model to obtain a noise map corresponding to the depth map, the method further includes: training the diffusion model based on a first image dataset; The step of training the diffusion model according to the first image data set comprises: Acquire the first image data set; the first image data set includes a depth map sequence and a color map sequence corresponding to the depth map sequence; Setting a first training prompt word for each of the depth maps in the depth map sequence; the first training prompt word is used to describe the category of each object and / or the position of each object in the depth map; Encoding the first training prompt word of each of the depth images by a text encoder to obtain a text feature vector of each of the depth images; The initial diffusion model is trained according to the depth map sequence, the color map sequence and the text feature vectors of each depth map until the training is completed, and the parameters of the diffusion model are stored to obtain the trained diffusion model.

4. The method according to claim 1, characterized in that The color video generation model includes a diffusion encoder and a diffusion decoder; The step of inputting the video to be processed into a color video generation model to generate a corresponding color video, and introducing a noise map encoded by a noise encoder to perform global semantic control on the color video generation process to obtain a color video includes: Based on the video to be processed, the depth map and its corresponding noise map are obtained, and a second text prompt word set for the target scene where the video to be processed is located is obtained; the second text prompt word is used to describe the target scene; Inputting the depth map and the noise map encoded by the noise encoder into the diffusion encoder for encoding to obtain an input vector corresponding to the depth map; Decoding the input vector and the second text prompt word by a diffusion decoder to obtain an output vector corresponding to the depth map, and performing secondary decoding on the output vector to obtain a color map corresponding to the depth map; The color images corresponding to the depth images are subjected to video synthesis processing to obtain the color video.

5. The method according to claim 1, characterized in that Before the video to be processed is input into the color video generation model to generate the corresponding color video, and the noise map encoded by the noise encoder is introduced to perform global semantic control on the color video generation process to obtain the color video, the method further includes: Acquire a second image data set to train a machine learning model until the training is completed, store the parameters of the machine learning model, and obtain an initial color video generation model, wherein the initial color video generation model has the ability to convert the video to be processed into the color video; Acquire second training prompt words set for a plurality of target scenes; the second training prompt words are used to describe the target scenes; The initial color video generation model is trained for a second time according to the acquired second text prompt word until the training is completed, and the parameters of the color video generation model are stored to obtain the trained color video generation model.

6. The method according to claim 4, characterized in that The step of performing video synthesis processing on the color images corresponding to the depth images to obtain the color video includes: Performing interval sampling on the color images corresponding to the depth images to obtain a plurality of key frames; Grouping the plurality of key frames to obtain a plurality of key frame sequences; the key frame sequences include at least one key frame; At least one of the key frames in each of the key frame sequences is fused to obtain the color video.

7. The method according to claim 6, characterized in that The step of fusing the key frames in each of the key frame sequences to obtain the color video comprises: For each of the key frame sequences, the key frames in the key frame sequence are spliced ​​to obtain a multi-frame fused image; the fused image corresponds to the key frame sequence one by one; The color video is obtained by performing cross-attention calculation of key frames on multiple frames of the fused image through an attention mechanism.

8. The method according to any one of claims 1 to 7, characterized in that After obtaining the color video, the method further includes: A semantic segmentation task is performed according to the color video to obtain a semantic segmentation result; the semantic segmentation result includes one or more of the position of the object and the category of the object in the target scene.

9. A color video generating device, characterized in that: The device comprises: A to-be-processed video acquisition module, used to acquire a to-be-processed video, wherein the to-be-processed video includes at least one depth map; A noise diffusion module, for each of the depth maps, performing noise diffusion on the depth map through a diffusion model to obtain a noise map corresponding to the depth map; the noise map is used to indicate semantic information and depth information of the depth map; The color video generation module is used to input the video to be processed into the color video generation model to generate the corresponding color video, and introduce the noise map encoded by the noise encoder to perform global semantic control on the color video generation process to obtain the color video.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the color video generation method according to any one of claims 1 to 8.

Images