A picture generation method and device, electronic equipment and product

Abstract

The application discloses a picture generation method and device, a storage medium and a product, relates to the technical field of virtual reality, and comprises the following steps: acquiring a virtual picture corresponding to a target virtual reality picture and a real object; determining a content adaptation degree of the real object and at least one picture segmentation area in the virtual picture, and a picture brightness of the at least one picture segmentation area; determining a target rendering area corresponding to the real object from the at least one picture segmentation area according to the picture brightness and the content adaptation degree; and rendering the real object in the target rendering area to generate the target virtual reality picture. By taking the picture brightness into the judgment basis of area matching, the matching precision of the real object and the target rendering area can be effectively improved, the situation that the real object is offset and dislocated in the rendering position is reduced, the fusion effect of the real object and the virtual scene is more natural and coordinated, and the visual sense of the virtual-real fusion picture and the user watching immersion experience are improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a method, apparatus, storage medium and product for generating images. Background Technology

[0002] In virtual reality panoramic video playback scenarios, it is often necessary to overlay and render objects detected in the real environment onto the virtual panoramic image to achieve a virtual-real fusion display effect.

[0003] Existing virtual-real fusion solutions rely solely on basic category matching or simple semantic matching to calculate the fit between objects and image regions, and then render the objects to the corresponding image regions to generate a virtual-real fusion rendering. However, this matching method has limited calculation accuracy, and the object rendering position is prone to offset and misalignment, resulting in a poor visual experience of the final virtual-real fusion image. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, storage medium and product for generating images, the main purpose of which is to solve the technical problem that the existing virtual-real fusion scheme has insufficient accuracy in matching calculation, and the object rendering position is prone to offset and misalignment, resulting in poor display effect of virtual-real fusion images.

[0005] Firstly, this application provides a method for generating an image, including: Acquire the virtual images and real-world objects corresponding to the target virtual reality image.

[0006] Determine the degree of content adaptation between the real object and at least one screen segmentation region in the virtual image, as well as the screen brightness of at least one screen segmentation region.

[0007] Based on the brightness of the image and the degree of content adaptation, the target rendering area corresponding to the real object is determined from at least one image segmentation area.

[0008] Real-world objects are rendered in the target rendering area to generate the target virtual reality image.

[0009] Secondly, this application provides an image generation apparatus, comprising: The acquisition module is configured to acquire the virtual image and real-world objects corresponding to the target virtual reality image.

[0010] The determination module is configured to determine the degree of content adaptation between the real object and at least one screen segmentation region in the virtual image, as well as the screen brightness of at least one screen segmentation region.

[0011] The matching module is configured to determine the target rendering area corresponding to the real object from at least one screen segmentation region based on screen brightness and content adaptation.

[0012] The rendering module is configured to render real-world objects in the target rendering area to generate the target virtual reality image.

[0013] Thirdly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the screen generation method described in the first aspect.

[0014] Fourthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the screen generation method described in the first aspect.

[0015] By employing the above technical solutions, this application provides a method, apparatus, storage medium, and product for generating images. Compared with existing technologies, this application first acquires the virtual image corresponding to the target virtual reality image and the real object to be superimposed, calculates the content adaptation degree between the real object and each image segmentation region, and extracts the image brightness corresponding to each image segmentation region. Subsequently, it comprehensively evaluates the content adaptation degree by combining the region brightness information, selects the target rendering region that adapts to the real object from each image segmentation region, and finally superimposes and renders the real object onto the target rendering region to output the target virtual reality image. By incorporating image brightness into the region matching criterion, this application can effectively improve the matching accuracy between the real object and the target rendering region, reduce the situation of real object rendering position offset misalignment, make the fusion effect of real objects and virtual scenes more natural and coordinated, and improve the visual perception of the virtual-real fusion image and the user's immersive viewing experience. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a screen generation method provided in an embodiment of this application is shown; Figure 2 This illustration shows a flowchart of a method for determining content adaptation degree according to an embodiment of this application; Figure 3This illustration shows a flowchart of a method for determining the target rendering region corresponding to a real-world object, according to an embodiment of this application. Figure 4 This illustration shows a schematic diagram of the structure of an image generation apparatus provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0020] To address the technical issues of insufficient accuracy in matching calculations in existing virtual-real fusion schemes, which easily lead to displacement and misalignment of object rendering positions and consequently poor display effects of virtual-real fusion images, this application selects suitable target rendering areas by comprehensively considering the content adaptation degree of each screen segmentation region and screen brightness, and performs superimposed rendering of real objects based on the target rendering areas. This can improve the matching accuracy between real objects and target rendering areas, reduce the displacement and misalignment of real object rendering positions, make the fusion effect of real objects and virtual scenes more natural and harmonious, and enhance the visual appeal and immersive viewing experience of virtual-real fusion images.

[0021] This embodiment provides a method for generating images, such as... Figure 1 As shown, the method includes: S101. Obtain the virtual image and real-world object corresponding to the target virtual reality image.

[0022] The target virtual reality (VR) visuals are panoramic videos of virtual reality (VR) and extended reality (XR) scenes, resulting in a fusion of virtual and real output. Extended reality includes mixed reality (MR), and this fusion output can be displayed through immersive terminals such as VR headsets and MR wearable devices.

[0023] The virtual image is a virtual scene image of a panoramic video frame currently being played on a display device (such as a VR or MR device). The virtual image is divided into several independent screen segments, each corresponding to fixed semantic tags such as sky, ground, buildings, trees, roads, and water surfaces.

[0024] The real objects referred to are real moving objects obtained by the display device after real-time acquisition of the real environment and target detection and keying processing, such as people, vehicles, birds and other real objects that need to be superimposed on the virtual screen.

[0025] In one implementation, a camera module mounted on a display device captures real-time images of the surrounding environment. For each captured frame of the environment image, a target detection model (such as the YOLOv5 model) is used to identify moving objects in the environment. Then, an edge detection algorithm (such as the Canney edge detection algorithm) is employed to extract the complete outline of the moving object. Based on the outline, the environment image is differentiated into foreground and background, irrelevant background elements are removed, and only the target object pixels are retained. This generates a transparent background image with a complete outline, resulting in a real-world object image without excessive background interference that can be directly overlaid and rendered.

[0026] On the other hand, for the panoramic virtual image currently being played in real time on the display device, a scene parsing network (such as a pyramid scene parsing network) is invoked to perform semantic segmentation processing on the virtual image. Based on the pixel features within the virtual image, the entire region is divided, and a single complete virtual image is segmented into multiple clearly defined, non-overlapping image segments. Then, combining the scene features within each region of the virtual image, a unique semantic label is matched to each image segment, such as sky, ground, buildings, trees, roads, and water. Simultaneously, key information such as scene attributes, pixel coordinate range, and area corresponding to each segment in the virtual image is fully recorded, and all structured region information is bound to the corresponding virtual image, enabling the virtual image to carry the semantic, positional, and size data of each image segment.

[0027] S102. Determine the degree of content adaptation between the real object and at least one screen segmentation region in the virtual image, as well as the screen brightness of at least one screen segmentation region.

[0028] The content adaptation degree is used to characterize the degree of fit and matching between real objects and corresponding screen segmentation areas in terms of scene content and scene logic. The higher the value, the stronger the rationality of the real object being adapted to the rendering and display within the screen segmentation area.

[0029] The image brightness is the overall illumination intensity value calculated by comprehensively analyzing all pixels within the corresponding image segmentation area, which can reflect the light and dark environment characteristics of that area.

[0030] In one implementation, for each screen segmentation region obtained by dividing the virtual screen, a comprehensive matching evaluation is carried out by combining the attribute information of the real object (such as object category, object size, motion attributes) with the key information of the screen segmentation region (such as scene attributes, pixel coordinate range, area) to quantitatively calculate the content adaptation degree between the real object and the screen segmentation region.

[0031] In another implementation, the RGB color pixel values ​​of each screen segmentation area are extracted, and the screen brightness corresponding to the screen segmentation area is obtained by weighted calculation based on the pixel components of each color channel.

[0032] S103. Based on the screen brightness and content adaptation, determine the target rendering area corresponding to the real object from at least one screen segmentation area.

[0033] The target rendering area is composed of one or more screen segmentation areas.

[0034] In one implementation, the brightness of the screen reflecting the ambient light and darkness of the area, as well as the degree of content adaptation between the segmented screen regions and the real-world objects, are used as dual evaluation criteria. A comprehensive adaptation evaluation is performed on all segmented screen regions one by one, and the target segmented screen region with the best overall adaptation effect is selected. If the target segmented screen region is large enough to completely accommodate the real-world object, it is directly determined as the target rendering region. If the target segmented screen region is insufficient to accommodate the real-world object, the size of the real-world object is scaled up to fit it. If the scaling still fails to accommodate the object, the target segmented screen region is merged with adjacent segmented screen regions to form the target rendering region.

[0035] For example, when the real-world object to be rendered is a pedestrian or vehicle, the target rendering area can be a ground or road segment that matches the scene logic and lighting environment. When the real-world object to be rendered is a bird or aircraft, a sky segment that has a higher matching degree can be selected as the target rendering area.

[0036] S104. Render the real object in the target rendering area to generate the target virtual reality screen.

[0037] In one implementation, the real-world object, obtained through keying, is superimposed onto the corresponding position of the target rendering area of ​​the virtual image for layered rendering. Then, the real-world object layer is merged and composited with the original panoramic virtual image layer to generate the target virtual reality image. This target virtual reality image is output through the display output module of a display device, allowing users to intuitively view the panoramic video content after the virtual-real fusion.

[0038] Compared with existing technologies, this application incorporates screen brightness into the criteria for region matching, which can effectively improve the matching accuracy between real objects and target rendering areas, reduce the situation of real object rendering position offset and misalignment, make the fusion effect of real objects and virtual scenes more natural and coordinated, and improve the visual perception of the virtual-real fusion screen and the user's immersive viewing experience.

[0039] The image generation method provided in the embodiments of this application will now be described in detail.

[0040] Optionally, this application also provides an embodiment for determining the content adaptation degree between a real object and at least one screen segmentation region in a virtual image, as well as the screen brightness of at least one screen segmentation region.

[0041] In this embodiment, the content adaptation degree is determined based on the common sense relationship matching degree between real objects and at least one screen segmentation region, as well as the semantic embedding similarity.

[0042] In one implementation, based on a pre-configured common-sense mapping relationship for scene categories, common-sense relationship matching is performed between the categories of real-world objects and the semantic tags of the segmented areas of the screen, and the corresponding common-sense relationship matching degree is output. For example, the preset common-sense mapping relationship defines that pedestrians, vehicles, and ground / road segmented areas have a reasonable coexistence relationship in real-world scenarios, and birds, aircraft, and sky segmented areas have a reasonable coexistence relationship. When there is no coexistence logic between the real-world object and the scene corresponding to the segmented area that conforms to everyday cognition, the common-sense relationship matching degree is set to a low value; when the scene logic matches, a high value is set.

[0043] In another implementation, semantic labels for the segmented image regions and semantic vectors corresponding to real-world objects are extracted, and semantic embedding similarity is obtained through vector similarity calculation. For example, a word vector model is used to convert the semantic label text of the segmented image regions and the category description of the real-world objects into semantic vectors with uniform dimensions. Then, cosine similarity is used to calculate the spatial distance between the two sets of vectors. The closer the spatial distance, the stronger the semantic association between the real-world object and the segmented image regions, and the higher the output semantic embedding similarity value. Conversely, the greater the spatial distance, the weaker the semantic association between the real-world object and the segmented image regions, and the lower the output semantic embedding similarity value.

[0044] After obtaining the common sense relationship matching degree and semantic embedding similarity, the common sense relationship matching degree and semantic embedding similarity are weighted and fused according to the preset weight coefficient to calculate the degree of content adaptation between the real object and the segmented area of ​​the image.

[0045] In this embodiment, the screen brightness of at least one screen segmentation region is determined based on the pixel value information of at least one screen segmentation region.

[0046] In one implementation, the RGB pixel values ​​corresponding to all pixels within a single image segmentation area are extracted. A standard brightness conversion formula is used to weight the red (R), green (G), and blue (B) channel components, and the results are normalized to obtain a uniform image brightness value for the segmented area, thus representing the overall brightness environment of the corresponding region. For example, fixed weighting coefficients can be used to perform brightness conversion for the R, G, and B channels respectively, mapping the calculation results to a value range of 0 to 1, facilitating subsequent matching correction and object transparency adjustment. The image brightness L... bg The calculation formula is shown below: ; The brightness of the entire image segmentation area is obtained by calculating the brightness of each pixel in the image segmentation area one by one using the above formula and taking the average value.

[0047] This embodiment combines common sense logic matching and semantic vector similarity to quantify the content adaptation degree. At the same time, it relies on RGB pixels to accurately calculate the brightness of the area screen. This not only ensures that the scene matching conforms to real cognition, but also provides reliable quantitative parameters for subsequent light and shadow fusion, effectively improving the rationality of the superposition of virtual and real objects and the realism of the screen fusion.

[0048] Optionally, this application also provides an embodiment for determining the content adaptation degree based on the common-sense relationship matching degree between real objects and at least one segmented area of ​​the image, as well as semantic embedding similarity. For example... Figure 2 The diagram illustrates a method for determining content adaptation, which includes the following embodiments: S201. Based on the pre-configured scene category common sense mapping relationship, perform common sense relationship matching mapping on real objects and at least one screen segmentation region to obtain the common sense relationship matching degree.

[0049] Optionally, the common sense relationship matching degree includes a first value and a second value. The first value indicates that there is a common sense relationship between the real object and at least one screen segmentation region, and the second value indicates that there is no common sense relationship between the real object and at least one screen segmentation region.

[0050] In one implementation, a binary matching matrix S based on scene common sense is pre-constructed. normal (R) i C j ), where i is the index of the image segmentation region, corresponding to the semantic label of the i-th image segmentation region. j is the category index of the real object, corresponding to the category label of the j-th real object category. Matrix S normal (R) i C j The semantic label R representing the i-th frame segmentation region i Category label C of the j-th type of real-world object jThe common sense matching score between them. This matching score is set to a first value of 1 and a second value of 0; when S normal (R) i C j When S = 0, it means that the j-th type of real-world object cannot be placed within the i-th screen segmentation area. normal (R) i C j When ) = 1, it means that the j-th type of real-world object can be placed in the i-th screen segmentation area.

[0051] For example, if the semantic label of the segmented area is ground, the common sense matching score of the area with the two object categories of people and vehicles is 1; if the semantic label of the segmented area is sky, the matching score of the area with the two object categories of birds and aircraft is 1, and the common sense matching score with people and vehicles is 0.

[0052] S202. Based on the semantic labels of at least one segmented area of ​​the image and the semantic vectors corresponding to the real objects, semantic embedding similarity is obtained through similarity calculation.

[0053] In one implementation, a semantic embedding model (such as Bert or Word2Vec) is used to embed the category label R corresponding to the i-th segmented image region. i Category label C of the j-th type of real-world object j Each is mapped to a high-dimensional semantic vector with the same dimension, and then the semantic embedding similarity S between the two is calculated using the cosine similarity formula. embed (R) i C j Its semantic embedding similarity S embed (R) i C j The formula for calculating () is shown below: ; Among them, R i ·C j This is the dot product operation for two sets of vectors; These are the magnitudes of the two sets of vectors, respectively.

[0054] S embed (R) i C j The value range of ) is [-1,1]; the higher the value, the higher the semantic association between the i-th image segmentation region and the j-th type of real object; conversely, the lower the value, the lower the semantic association between the two.

[0055] S203. Based on the common sense relationship matching degree and semantic embedding similarity, the content adaptation degree is obtained.

[0056] In one implementation, common sense relationship matching degree is used as a pre-screening condition. Only when the common sense relationship matching degree is 1, the current screen segmentation region is retained. Then, the content adaptation degree between the real object and the screen segmentation region is obtained based on semantic embedding similarity. If the common sense relationship matching degree is 0, the current screen segmentation region is directly discarded.

[0057] In another implementation, fixed weight coefficients are configured for the two sets of indicators. The common sense relationship matching degree is multiplied by a first weight, and the semantic embedding similarity degree is multiplied by a second weight. The products of the two are then summed to obtain the quantified content fit. For example, the first weight of the common sense relationship matching degree can be set to 0.3, and the second weight of the semantic embedding similarity degree can be set to 0.7 to complete the weighted summation of the two indicators.

[0058] This embodiment first relies on common sense binary judgment to complete the coarse screening, eliminating invalid areas that violate the logic of the real scene, and then uses semantic vector similarity to quantify the matching degree. It takes into account both the scene common sense constraints and the semantic association for fine scoring, effectively narrowing the candidate range and improving the rationality of the object placement area selection, thus enhancing the naturalness of the virtual-real fusion picture.

[0059] Optionally, this application also provides an embodiment for determining the target rendering area corresponding to a real-world object from at least one screen segmentation region based on screen brightness and content adaptation. For example... Figure 3 The diagram illustrates a method for determining the target rendering region corresponding to a real-world object. This embodiment includes: S301. Determine the illumination correction factor based on the screen brightness.

[0060] In one implementation, the image brightness L of the i-th image segmentation region is extracted, and the brightness and darkness intervals are divided based on a preset brightness threshold to generate a corresponding illumination correction factor S. L For example, a brightness threshold of 200 lux is set, and a segmented correction rule is configured: if the screen brightness L is greater than 200 lux, then the illumination correction factor S... L The value is +0.2. If the screen brightness L is less than or equal to 200 lux, then the illumination correction factor S is set to +0.2. L The value is -0.1. This correction factor is used to characterize the degree of gain or loss in the reasonableness of the matching of the regional light and dark environment to the real objects.

[0061] S302. Use the illumination correction factor to perform weighted correction on the content adaptation degree to obtain the target matching degree corresponding to each screen segmentation area.

[0062] In one implementation, a weighted fusion calculation is performed using a comprehensive matching degree formula. The comprehensive matching degree sim(Ri, Cj) is the target matching degree after illumination correction, and its calculation formula is shown below: ; In the formula, α represents the semantic embedding similarity S. embed (R) i C j The weighting coefficients of ), where β is the illumination correction factor S. L The weighting coefficients. Both have values ​​ranging from 0 to 1 and can be pre-configured according to scene rendering requirements.

[0063] The entire screen segmentation region is traversed and combined with the real object to generate a two-dimensional matching degree matrix L[N][K]. Each item in the matrix corresponds to the target matching degree between a set of screen segmentation regions and real objects.

[0064] S303. Based on the target matching degree, determine the target rendering area corresponding to the real object from at least one screen segmentation area.

[0065] In one implementation, for the j-th type of real-world object, the target matching degree values ​​of all corresponding regions in the matching degree matrix L[N][K] are traversed, and the i-th image segmentation region with the largest value is selected. This region is initially determined as the target image segmentation region that is suitable for the real-world object. Subsequently, based on the size relationship between the real-world object and this region, scaling and region merging processes are performed to finally determine the complete target rendering region.

[0066] This embodiment introduces a regional brightness generation illumination correction factor to dynamically compensate the basic matching results. It incorporates three dimensions—scene semantics, common sense logic, and lighting environment—into the matching evaluation, which greatly improves the accuracy of regional selection, avoids object matching distortion in bright and dark scenes, and makes the superposition of virtual and real objects more consistent with the real lighting logic, further enhancing the realistic visual experience of the merged image.

[0067] Optionally, this application also provides an embodiment for determining the target rendering region corresponding to a real-world object from at least one screen segmentation region based on the degree of target matching. In this embodiment, the screen segmentation region with the highest target matching degree with the real-world object is determined as the target screen segmentation region; if the size of the real-world object is larger than the target screen segmentation region, the real-world object is scaled according to a preset scaling factor; if the size of the scaled real-world object is larger than the target screen segmentation region, the target screen segmentation region and adjacent screen segmentation regions are merged; the screen segmentation region obtained after scaling and / or merging is determined as the target rendering region.

[0068] In one implementation, the image segmentation region that best matches the real-world object is determined as the target image segmentation region. The bounding box area of ​​the real-world object C is defined as Size(C), and the area of ​​the target image segmentation region R is defined as Size(R). The ratio of the two is... It represents the dimensional ratio between an object and a region.

[0069] When sim size (R) i C j When ) is greater than 1, it indicates that the area of ​​the real object is larger than the area of ​​the target image segmentation region; when sim size (R) i C j When 1 is less than or equal to 1, it means that the area of ​​the real object is less than or equal to the area of ​​the target image segmentation region.

[0070] When the area of ​​a real-world object is larger than the area of ​​the target image segmentation region, the scaling factor Scale(C) is calculated according to the following piecewise formula, and scaling processing is performed on the real-world object: ; In the formula, when the area of ​​the bounding box of the real object C, Size(C), is greater than the area of ​​the target image segmentation region R, Size(R), the square root of the area ratio is used as the scaling factor Scale(C) to proportionally reduce the size of the real object. When the area of ​​the real object is not greater than the area of ​​the region (other), the scaling factor Scale(C) is fixed at 1 to keep the original size of the object unchanged.

[0071] In one implementation, if the size of the real-world object still exceeds the target image segmentation area after scaling, the target image segmentation area is merged with its adjacent image segmentation areas to form a continuous extended area. The image segmentation area obtained after scaling and / or merging is ultimately determined as the target rendering area for the real-world object.

[0072] This embodiment first selects the optimal semantic region based on the target matching degree, and then dynamically calculates the scaling factor through the area ratio, scaling down only for very large objects to avoid image magnification distortion. When a single scaling is still insufficient to accommodate the object, adjacent regions are automatically merged to expand the rendering range, so that the size of the real object and the virtual background area are adaptively matched, effectively solving the problems of objects going out of bounds and disproportionate when the virtual and real are superimposed, and improving the layout rationality of the panoramic video overlay.

[0073] Optionally, after merging the target image segmentation region and adjacent image segmentation regions, this application also provides an embodiment for determining the transparency rendering parameters of a real-world object. In this embodiment, based on the image brightness of each basic image segmentation region included in the merging process, the basic transparency corresponding to each basic image segmentation region is obtained; based on the basic transparency corresponding to each basic image segmentation region, the local transparency rendering parameters of the real-world object within each basic image segmentation region are determined; and based on the local transparency rendering parameters of the real-world object, the global transparency rendering parameters of the real-world object are obtained.

[0074] In one implementation, the RGB pixel values ​​corresponding to each pixel point (x,y) covered by the real object are extracted, and the image brightness is calculated and normalized to the [0,1] interval. Based on the mapping relationship between image brightness and transparency, the basic transparency corresponding to each basic image segmentation region is generated. The brightness value L... bg The closer a value is to 1, the brighter the background environment and the higher the opacity of the corresponding local area of ​​the real object. Brightness value L bg The closer a value is to 0, the darker the background environment and the higher the transparency of the corresponding local area of ​​the real object.

[0075] In scenarios where a real-world object spans multiple basic image segmentation regions, the area coverage ratio of the real-world object within each basic image segmentation region is statistically analyzed. The local transparency is then weighted using the coverage ratio as the weight, and the global transparency rendering parameters of the real-world object are finally obtained.

[0076] For example, the calculation method for the weighted average of local transparency is shown in the following formula: ; in, Represents the basic screen segmentation region R i quantity; W i For the real-world object C, segment the basic image area R. i The coverage percentage in; T(x, y) represents the base opacity of the real-world object pixel (x, y) within a single region, with a value range of [0, 1], and is determined by the normalized brightness L of the background region. bg The calculation yields T(x, y) = 1 - L bg That is, L bg The larger the value, the higher the basic opacity.

[0077] T i (x, y) represents the location of the object within the i-th segmented region R. i Internal base opacity; T min (x, y) represents the percentage W of an object's coverage across multiple regions when the object is displayed across those regions.i The pixel blending opacity obtained by weighted calculation is used to eliminate the sense of disjointed blending caused by brightness differences across regions.

[0078] In one implementation, the local transparency rendering parameters of the real-world object within each basic frame segmentation area are aggregated to form a transparency parameter set, which serves as the global transparency parameter set for the entire object (i.e., the global transparency rendering parameters). Then, a dual-channel separate rendering scheme is used to complete texture compositing: first, the virtual image is rendered to a first frame buffer object (FBO); then, the real-world object with foreground keying is rendered region by region to an independent second frame buffer object (FBO) using the local transparency and lighting parameters of the corresponding blocks in this transparency parameter set. Finally, the textures in the two sets of frame buffer objects are blended pixel by pixel, and the foreground and background are superimposed and blended according to the partitioned transparency to output the target virtual reality image.

[0079] Compared with existing technologies, this application incorporates screen brightness into the criteria for region matching, which can effectively improve the matching accuracy between real objects and target rendering areas, reduce the situation of real object rendering position offset and misalignment, make the fusion effect of real objects and virtual scenes more natural and coordinated, and improve the visual perception of the virtual-real fusion screen and the user's immersive viewing experience.

[0080] Furthermore, as Figures 1 to 3 To illustrate the specific implementation of the method shown, this embodiment provides a screen generation device, such as... Figure 4 The diagram shows a structural schematic of an image generation device 400, which includes an acquisition module 410, a determination module 420, a matching module 430, and a rendering module 440, wherein: The acquisition module 410 is configured to acquire the virtual image and real object corresponding to the target virtual reality image.

[0081] The determining module 420 is configured to determine the degree of content adaptation between the real object and at least one screen segmentation region in the virtual image, as well as the screen brightness of at least one screen segmentation region.

[0082] Matching module 430 is configured to determine the target rendering area corresponding to the real object from at least one screen segmentation area based on screen brightness and content adaptation.

[0083] The rendering module 440 is configured to render real-world objects in the target rendering area to generate a target virtual reality image.

[0084] Optionally, when determining the content adaptation degree between the real object and at least one screen segmentation region in the virtual image, and the screen brightness of at least one screen segmentation region, the determining module 420 is further configured to: The degree of content adaptation is determined based on the common-sense relationship matching degree between real objects and at least one segmented area of ​​the screen, as well as the semantic embedding similarity.

[0085] The brightness of at least one screen segmentation region is determined based on the pixel value information of at least one screen segmentation region.

[0086] Optionally, when determining the content adaptation degree based on the common-sense relationship matching degree between real objects and at least one segmented area of ​​the screen, and the semantic embedding similarity, the determining module 420 is further configured to: Based on the pre-configured common sense mapping relationship of scene categories, common sense relationship matching and mapping is performed on real objects and at least one screen segmentation region to obtain the common sense relationship matching degree.

[0087] Based on the semantic labels of at least one segmented area of ​​the image and the semantic vectors corresponding to the real objects, semantic embedding similarity is obtained through similarity calculation.

[0088] The content fit is determined by the common sense relationship matching degree and semantic embedding similarity.

[0089] Optionally, the common sense relationship matching degree includes a first value and a second value. The first value indicates that there is a common sense relationship between the real object and at least one screen segmentation region, and the second value indicates that there is no common sense relationship between the real object and at least one screen segmentation region.

[0090] Optionally, when determining the target rendering area corresponding to the real object from at least one screen segmentation region based on screen brightness and content adaptation, the matching module 430 is further configured to: The illumination correction factor is determined based on the image brightness.

[0091] By using a lighting correction factor to weight and correct the content adaptation degree, the target matching degree corresponding to each segmented area of ​​the screen is obtained.

[0092] Based on the degree of target matching, determine the target rendering area corresponding to the real object from at least one screen segmentation region.

[0093] Optionally, when determining the target rendering region corresponding to the real object from at least one screen segmentation region based on the degree of target matching, the matching module 430 is further configured to: The image segmentation region that best matches the real-world object is defined as the target image segmentation region.

[0094] If the size of the real-world object is larger than the target image segmentation area, the real-world object is scaled according to a preset scaling factor.

[0095] If the size of the scaled real-world object is larger than the target image segmentation area, the target image segmentation area and adjacent image segmentation areas are merged.

[0096] The segmented area of ​​the image obtained after scaling and / or merging is determined as the target rendering area.

[0097] Optionally, after merging the target image segmentation region and adjacent image segmentation regions, the matching module 430 is further configured to: Based on the brightness of each basic image segmentation region included in the merging process, the basic transparency of each basic image segmentation region is obtained. Each basic image segmentation region includes the target image segmentation region and the adjacent image segmentation region.

[0098] Based on the base transparency corresponding to each basic image segmentation region, the local transparency rendering parameters of the real object within each basic image segmentation region are determined.

[0099] Based on the local transparency rendering parameters of the real object, the global transparency rendering parameters of the real object are obtained.

[0100] It should be noted that other corresponding descriptions of the functional units involved in the image generation apparatus provided in this embodiment can be found in [reference needed]. Figures 1 to 3 The corresponding descriptions in [the document] will not be repeated here.

[0101] Based on the above, Figures 1 to 3 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figures 1 to 3 The method shown.

[0102] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0103] like Figure 5 The diagram shown is a hardware structure schematic of an electronic device according to the present invention, comprising: At least one processor 501; and, A memory 502 is communicatively connected to at least one of the processors 501; wherein, The memory 502 stores instructions that can be executed by at least one of the processors to enable at least one of the processors to perform the image generation method as described above.

[0104] Figure 5 Take a processor 501 as an example.

[0105] The electronic device may also include an input device 503 and a display device 504.

[0106] The processor 501, memory 502, input device 503, and display device 504 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0107] The memory 502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the screen generation method in the embodiments of this application, for example, Figures 1 to 3 The method flow is shown. The processor 501 executes various functional applications and data processing by running non-volatile software programs, instructions, and modules stored in the memory 502, thereby implementing the screen generation method in the above embodiments.

[0108] The memory 502 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the screen generation method, etc. Furthermore, the memory 502 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 502 may optionally include memory remotely located relative to the processor 501, and these remote memories can be connected to the apparatus performing the screen generation method via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The input device 503 can receive user clicks and generate signal inputs related to user settings and function control of the screen generation method. The display device 504 may include a display screen or other display device.

[0110] When one or more modules are stored in the memory 502, and are run by one or more processors 501, the screen generation method in any of the above method embodiments is executed.

[0111] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0112] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0113] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms, or it can be implemented by hardware. By applying the solution of this embodiment, compared with the prior art, this application, by incorporating screen brightness into the determination criteria for region matching, can effectively improve the matching accuracy between real objects and target rendering areas, reduce the situation of real object rendering position offset misalignment, make the fusion effect of real objects and virtual scenes more natural and coordinated, and improve the visual perception of virtual-real fusion images and the user's immersive viewing experience.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating an image, characterized in that, include: Acquire the virtual images and real-world objects corresponding to the target virtual reality image; Determine the degree of content adaptation between the real object and at least one screen segmentation region in the virtual image, as well as the screen brightness of the at least one screen segmentation region; Based on the image brightness and the content adaptation degree, the target rendering area corresponding to the real object is determined from the at least one image segmentation area; The real-world object is rendered in the target rendering area to generate the target virtual reality image.

2. The method according to claim 1, characterized in that, Determining the content adaptation degree between the real object and at least one screen segmentation region in the virtual image, and the screen brightness of the at least one screen segmentation region, includes: The content adaptation degree is determined based on the common sense relationship matching degree between the real object and the at least one screen segmentation region, as well as the semantic embedding similarity. The brightness of the at least one screen segmentation region is determined based on the pixel value information of the at least one screen segmentation region.

3. The method according to claim 2, characterized in that, The determination of the content adaptation degree based on the common-sense relationship matching degree between the real object and the at least one segmented image region, and the semantic embedding similarity, includes: Based on the pre-configured scene category common sense mapping relationship, common sense relationship matching and mapping are performed on the real object and the at least one screen segmentation region to obtain the common sense relationship matching degree; Based on the semantic labels of the at least one image segmentation region and the semantic vectors corresponding to the real-world objects, the semantic embedding similarity is obtained through similarity calculation; The content fit is obtained based on the common sense relationship matching degree and the semantic embedding similarity.

4. The method according to claim 3, characterized in that, The common sense relationship matching degree includes a first value and a second value. The first value indicates that there is a common sense relationship between the real object and the at least one screen segmentation region, and the second value indicates that there is no common sense relationship between the real object and the at least one screen segmentation region.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the target rendering area corresponding to the real object from the at least one screen segmentation area based on the screen brightness and the content adaptation degree includes: The illumination correction factor is determined based on the image brightness. The content adaptation degree is weighted and corrected using the illumination correction factor to obtain the target matching degree corresponding to each screen segmentation region. Based on the target matching degree, the target rendering area corresponding to the real object is determined from the at least one screen segmentation area.

6. The method according to claim 5, characterized in that, Determining the target rendering region corresponding to the real-world object from the at least one screen segmentation region based on the target matching degree includes: The image segmentation region that best matches the real-world object is defined as the target image segmentation region. If the size of the real object is larger than the target image segmentation area, the real object is scaled according to a preset scaling factor. If the size of the scaled real-world object is larger than the target image segmentation region, the target image segmentation region and adjacent image segmentation regions are merged. The image segmentation region obtained after scaling and / or merging is determined as the target rendering region.

7. The method according to claim 6, characterized in that, After merging the target image segmentation region and adjacent image segmentation regions, the method further includes: Based on the brightness of each basic screen segmentation region included in the merging process, the basic transparency corresponding to each basic screen segmentation region is obtained. Each basic screen segmentation region includes the target screen segmentation region and adjacent screen segmentation regions. Based on the basic transparency corresponding to each of the basic image segmentation regions, the local transparency rendering parameters of the real object within each of the basic image segmentation regions are determined; Based on the local transparency rendering parameters of the real object, the global transparency rendering parameters of the real object are obtained.

8. An image generation device, characterized in that, include: The acquisition module is configured to acquire the virtual image and real-world objects corresponding to the target virtual reality image. The determining module is configured to determine the degree of content adaptation between the real object and at least one screen segmentation region in the virtual image, as well as the screen brightness of the at least one screen segmentation region; The matching module is configured to determine the target rendering area corresponding to the real object from the at least one screen segmentation area based on the screen brightness and the content adaptation degree. The rendering module is configured to render the real-world object in the target rendering area to generate the target virtual reality image.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images