Projection image processing method and device based on augmented reality, terminal and medium

Abstract

The invention relates to the technical field of image processing, and discloses a projection image processing method and device based on reality augmentation, a terminal and a medium, and the method comprises the steps: obtaining current projection space information, current environment state information and to-be-projected optical related information; determining reference projection image information according to the current projection space information and the to-be-projected optical related information; determining reference virtual element image information according to the current environment state information and the reference projection image information; performing reality augmentation processing on the reference virtual element image information and the reference projection image information to obtain target projection image information; and performing projection display on the target projection image information in the current projection space and the current environment state. By implementing the method provided by the invention, the effectiveness of projection display can be improved, and the purpose of more effective, richer and more interesting projection display can be realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a projection image processing method, device, terminal and medium based on reality augmentation. Background Art

[0002] With the popularization of projection scenes, traditional projection methods are gradually unable to meet the audience's requirements for projection experience. At present, it is usually a simple projecting of preset images or video content into the corresponding projection space, which can only achieve the most basic picture display effect. The presented content is mostly static and lacks change, which makes it difficult to attract the audience's attention for a long time, and it is even more difficult to fully mobilize the audience's interest points, making the entire projection process boring. The projection scene cannot make the audience feel immersive. There is an obvious sense of separation between the projection picture and the real environment. The audience is only watching the picture as a bystander and cannot really integrate into the displayed scene, which makes the effectiveness of the entire projection display low. Therefore, how to improve the effectiveness of projection display and provide the audience with more effective, richer and more interesting projection displays has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides a projection image processing method, device, terminal and medium based on reality augmentation, which can improve the effectiveness of projection display and help achieve the purpose of more effective, richer and more interesting projection display.

[0004] In a first aspect, a projection image processing method based on reality augmentation is provided, comprising: Obtain current projection space information, current environment status information and optical related information to be projected; Determine reference projection image information according to current projection space information and optical related information to be projected; Determine reference virtual element image information according to current environment state information and reference projection image information; Performing reality augmentation processing on the reference virtual element image information and the reference projection image information to obtain the target projection image information; Under the current projection space and current environment status, the target projection image information is projected and displayed.

[0005] In a second aspect, a projection image processing device based on augmented reality is provided, comprising: An acquisition module is used to acquire current projection space information, current environment status information and optical related information to be projected; A first determination module, configured to determine reference projection image information according to the current projection space information and the optical related information to be projected; A second determining module, configured to determine reference virtual element image information according to the current environment state information and the reference projection image information; A processing module, used for performing reality enhancement processing on the reference virtual element image information and the reference projection image information to obtain target projection image information; The display module is used to project and display the target projection image information in the current projection space and the current environment state.

[0006] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned projection image processing method based on augmented reality when executing the computer program.

[0007] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned projection image processing method based on augmented reality are implemented.

[0008] In the scheme implemented by the above-mentioned projection image processing method, device, terminal and medium based on reality enhancement, by obtaining the current projection space information, the current environmental state information and the optical-related information to be projected, the reference projection image information can be determined according to the current projection space information and the optical-related information to be projected, and the reference virtual element image information can be determined according to the current environmental state information and the reference projection image information, so that the reference virtual element image information and the reference projection image information can be subjected to reality enhancement processing to obtain the target projection image information, and then the target projection image information can be projected and displayed in the current projection space and the current environmental state, which can improve the effectiveness of the projection display and is conducive to achieving the purpose of more effective, richer and more interesting projection display. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0010] Figure 1 is a flow chart of a projection image processing method based on reality augmentation in one embodiment of the present invention; Figure 2 is a structural schematic diagram of a projection image processing device based on reality augmentation in one embodiment of the present invention; Figure 3is a schematic diagram of a structure of a computer device in one embodiment of the present invention; Figure 4 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] In order to better understand the projection image processing method based on reality enhancement provided in the embodiment of the present application, the following first briefly introduces the scenario in which the projection image processing method based on reality enhancement is applied. At present, more and more scenarios can use projection display technology to enrich the projection display content, for example, exhibition display scenarios, such as museums, art exhibition halls and other exhibition venues to display various exhibits and artworks; education and training scenarios, such as classroom teaching or professional skills training; business activities, such as new product launches or brand display activities; interior decoration scenes, such as interior decoration of commercial places or home decoration; theme entertainment scenes, such as theme parks, virtual reality experience halls, etc., can all use projection display technology to better highlight the items to be displayed.

[0013] However, the existing projection display simply projects a preset image on the object to be displayed, or simply repeatedly projects the corresponding projection video content of the object to be displayed in the projection space, which cannot immerse the audience in the object or scene to be displayed. For example, in the process of displaying artworks, the existing projection technology can simply statically project the use scene of the artwork so that the artwork can be displayed in a similar way to a specific use scene. However, if there is a huge difference between the use scene (i.e., the projection picture) and the real environment, the audience will not be able to integrate into the display scene, thereby losing the role of the projection display and reducing the effectiveness of the projection display.

[0014] In order to solve the above problems, the embodiment of the present application provides a projection image processing method based on augmented reality, which can better adapt a reference projection image suitable for the current projection space according to the current projection space information and the optical-related information to be projected, and can determine a more interactive and immersive reference virtual element according to the current environmental state information and the reference projection image, and can further perform reality augmentation processing on the reference virtual element and the reference projection image to obtain a more integrated target projection image, so that the audience can better integrate into the display scene and get a better viewing experience, such as a more real understanding of the displayed artwork, or a more intuitive understanding of the teaching knowledge points, etc., thereby making the projection display process more effective.

[0015] See also Figure 1 As shown, Figure 1 A flowchart of a projection image processing method based on reality enhancement provided by an embodiment of the present invention includes the following steps: S10: Acquire current projection space information, current environment status information and optical related information to be projected.

[0016] The current projection space information may be used to indicate the space-related attribute description information where the current projection display behavior occurs. The current projection space information may include but is not limited to the current space size, current space shape, space background material and background color, etc., and this application does not limit this.

[0017] The current environment state information may be a comprehensive description of the real environment state of the current projection display behavior. The current environment state information may include, but is not limited to, the current environment atmosphere state information, such as light, temperature, humidity, sound, etc., and the current environment physical state information, such as the layout state information of temporary obstacles or the layout state of fixed physical objects, etc., which is not limited in this application.

[0018] The optically relevant information to be projected can be used to indicate the optically relevant information of the original content to be projected, such as the holographic projection light wave information to be projected, etc. The optically relevant information to be projected can include appearance feature information such as the shape, color, texture, etc. of the object or scene to be projected, and can be converted into a visible projection picture through specific optical processing and projection equipment.

[0019] Specifically, the above-mentioned current projection space information and current environmental state information can be obtained through preset sensors, such as a depth sensor to obtain the current projection space information, and a temperature sensor, a humidity sensor, etc. to obtain the current environmental state information. Optionally, computer vision technology can be used, such as using two cameras to capture images of the projection space, to further calculate and obtain the current projection space information and the current environmental state information by matching feature points in the left and right images. Optionally, the optically related information to be projected can be pre-acquired. For example, if the optically related information to be projected is holographic projection light wave information, the holographic projection light wave information can be obtained by pre-holographically photographing the content to be projected and displayed. This application does not limit this.

[0020] S20: Determine reference projection image information according to the current projection space information and the optical related information to be projected.

[0021] The reference projection image information may be information related to the projection form and projection content preliminarily presented in the current spatial environment, determined based on the current projection space information and the optical related information to be projected.

[0022] Specifically, the information related to the optics to be projected can be adjusted according to the size and shape of the projection space. For example, for a projection space in the form of a long and narrow corridor, the information related to the optics to be projected that is originally suitable for display in a square space can be stretched or cropped to obtain the above-mentioned reference projection image information, so that it can be fully and appropriately displayed in the projection space in the form of a long and narrow corridor.

[0023] Optionally, different projection devices and spatial distances may require projection images with different resolutions, color modes, and / or pixel depths to ensure visual effects. For example, for a projection display scene with a long spatial distance, a reference projection image information with a higher resolution may be set; for a projection display scene with a short spatial distance, a reference projection image information with a lower resolution may be set, and this application does not limit this.

[0024] S30: Determine reference virtual element image information according to the current environment state information and the reference projection image information.

[0025] The reference virtual element image information may be image data description information related to virtual elements that are determined based on the current environment state information and the reference projection image information and are used to be added to the reference projection image to achieve the reality augmentation effect. It is understandable that the reference virtual element image may be integrated with the reference projection image and the real environment (such as the current projection space and environment) to enhance the immersion and interactivity of the projection display process.

[0026] Optionally, the reference virtual element image information may include but is not limited to various virtual elements, such as virtual decorations, virtual natural scenery, etc., virtual light and shadow effects, such as simulated sunlight spots, lighting effects, shadows, lightning, etc., virtual humidity atmosphere elements, such as fog, snowflakes, smoke, etc., virtual sound atmosphere elements, such as wind, rain, thunder, etc., virtual interactive elements, such as interactive objects, clickable buttons, etc., and the present application does not impose any restrictions on this.

[0027] Optionally, for each virtual element, the reference virtual element image information may include a detailed description of the geometric shape, size, material texture, color appearance, etc. of each virtual element. For example, for a virtual stone bench element, the reference virtual element image information may include the length, width, height, stone material, texture style, color tone, etc. of the virtual stone bench. This application does not impose any restrictions on this.

[0028] Optionally, for virtual light and shadow, the reference virtual element image information may include information related to the light and shadow effects of virtual light and shadow under the current ambient lighting conditions, such as information on the illuminated surface, the backlit surface, the highlight position, the shadow projection effect, etc. Optionally, for virtual interactive elements, the reference virtual element image information may include information on the interaction logic between the virtual interactive elements and the user, such as the response effect after the user touches the virtual button, the performance effect when the virtual object collides or occludes the real object, etc., and this application does not limit this.

[0029] It should be understood that determining the reference virtual element image information according to the current environment state information and the reference projection image information refers to the process of how to determine the reference virtual element image information. In step S30, that is, determining the reference virtual element image information according to the current environment state information and the reference projection image information, includes the following steps: S31: acquiring current environment atmosphere state information from the current environment state information, and acquiring current environment physical state information from the current environment state information; S32: determining lighting atmosphere virtual element image information according to the current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information; S33: determining humidity atmosphere virtual element image information according to the current humidity atmosphere state information in the current environmental atmosphere state information and the reference projection image information; S34: Determine interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical state information; S35: Determine reference virtual element image information according to the lighting atmosphere virtual element image information, the humidity atmosphere virtual element image information and the interaction virtual element image information.

[0030] Among them, the current environment atmosphere state information can be used to describe the overall atmosphere of the environment in which the current projection behavior is located. The current environment atmosphere state information can integrate various factors in the environment, such as the state information of the relevant atmosphere created by light, humidity, temperature, sound, etc., and can provide a basis for the subsequent determination of reference virtual elements that match the atmosphere, so that the reference virtual elements can enhance the overall immersion when integrated into the current environment.

[0031] Specifically, the current environment atmosphere state information may include but is not limited to the lighting atmosphere, humidity atmosphere, temperature atmosphere and sound atmosphere, etc., and this application does not limit this. Optionally, the following text uses the current environment atmosphere state information including the current lighting atmosphere state information and the current humidity atmosphere state information as an example for explanation, which does not constitute a limitation on this application.

[0032] The lighting atmosphere virtual element image information may be determined based on the current lighting atmosphere state information and the reference projection image information, and is used to be added to the reference projection image to enhance the lighting-related atmosphere effect and make it more in line with the actual environment lighting feeling. The lighting virtual element image can simulate various effects produced by lighting, such as light spots, light shadows, reflected light, etc., and further correspond to the lighting conditions in the environment to create a more realistic and natural lighting environment.

[0033] It should be understood that determining the lighting atmosphere virtual element image information according to the current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information refers to the process of how to determine the lighting atmosphere virtual element image information. Among them, step S32, that is, determining the lighting atmosphere virtual element image information according to the current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information, includes the following steps: S321: Acquire the illumination intensity information of the current light source, the illumination direction information of the current light source, and the viewing angle depth map of the current light source from the current illumination atmosphere state information; S322: Determine reference virtual lighting direction information of a lighting atmosphere virtual element according to the lighting direction information of the current light source and preset virtual position information in the reference projection image information; S323: Determine reference virtual lighting intensity information of the lighting atmosphere virtual element according to the current light source lighting intensity information, the reference virtual lighting direction information and the diffuse reflection coefficient; S324: Determine reference virtual lighting shadow information of the lighting atmosphere virtual element according to the current light source viewing angle depth map; S325: Determine reference virtual lighting highlight information of the lighting atmosphere virtual element according to the highlight reflection coefficient, the highlight index, the sight line direction, the lighting direction information of the current light source and the preset virtual position information; S326: Determine lighting atmosphere virtual element image information according to the reference virtual lighting direction information, the reference virtual lighting intensity information, the reference virtual lighting shadow information and the reference virtual lighting highlight information.

[0034] It should be noted that the current lighting atmosphere state information may include but is not limited to the current light source illumination intensity information, the current light source illumination direction information, and the current light source viewing angle depth map, etc. Among them, the current light source illumination intensity information may be quantitative description information of the brightness of the light emitted by the light source in the current environment, which may reflect the overall lighting intensity of the environment, and may be used to determine the brightness of the reference virtual element in the reference projection image, as well as the coordination with the surrounding environment lighting.

[0035] The current light source illumination direction information can be used to describe the specific direction of the light source in the current environment, and can reflect the situation of light being emitted from the light source in various directions. It plays an important role in the subsequent determination of the light conditions of different parts of the reference virtual element surface and the overall light and shadow distribution. The current light source perspective depth map can be understood as the image information obtained by rendering the projected scene from the perspective of the light source. Each pixel in the image does not record conventional color information, but the distance information of the object surface from the light source in the corresponding scene, that is, the depth information. The current light source perspective depth map can be mainly used to determine the shadow situation of the virtual element in the lighting environment and the spatial occlusion relationship with other objects.

[0036] The reference virtual lighting direction information of the lighting atmosphere virtual element can be determined by comprehensively analyzing the lighting direction information of the current light source and the position information of the lighting atmosphere virtual element itself. It is a data description used to describe the actual lighting direction of the virtual element in the current lighting environment. It determines the lighting differences on different surfaces of the virtual element and the distribution direction of light and shadow.

[0037] The reference virtual lighting direction information of the lighting atmosphere virtual element can be determined by combining the lighting direction information of the current light source and the preset virtual position information in the reference projection image information (i.e., the preset position information of the lighting atmosphere virtual element), and is used to describe the direction of the actual lighting received by the reference virtual element in the current lighting environment. The reference virtual lighting direction information can determine the difference in lighting received at different positions on the surface of the virtual element and the distribution direction of light and shadow, and this application does not limit this.

[0038] It is understandable that the lighting atmosphere virtual element has its specific coordinate position and posture (such as rotation angle, etc.) in space, and can be combined with the current light source lighting direction information to determine the actual angle of light irradiation to each surface of the reference virtual element according to the light receiving principle of the lighting atmosphere virtual element in space. Specifically, taking the reference virtual element as a rectangular object, and its location diagonally below the light source as an example, the angle between the light and the normal direction of each face of the rectangular can be calculated to clarify the lighting direction received by each face, such as whether it is vertical illumination, oblique illumination, or in a backlit state, etc. The angle between the lighting direction and the normal direction of the virtual element surface can be represented and calculated through vector operations, so as to accurately describe the lighting situation of the virtual element.

[0039] Optionally, the process of determining the reference virtual lighting direction information of the lighting atmosphere virtual element according to the current light source lighting direction information and the preset virtual position information in the reference projection image information may refer to the following formula:

[0040] in, The reference virtual lighting direction information of the lighting atmosphere virtual element can be represented, that is, the angle between the lighting direction of the light source and the normal direction of the virtual element surface; It can represent the cosine information of the reference virtual lighting direction; It can represent the light source lighting direction vector in the current light source lighting direction information; A normal vector of a preset virtual element surface in preset virtual position information in the reference projection image information may be represented; It can be used to indicate the click operation between the light source illumination direction vector and the normal vector of the preset virtual element surface; Can be used to represent the magnitude of the light source's illumination direction vector; The magnitude of the normal vector that can be used to represent the surface of a preset virtual element.

[0041] The reference virtual lighting intensity information of the lighting atmosphere virtual element can be calculated based on the current light source lighting intensity information, the reference virtual lighting direction information and the diffuse reflection coefficient, and is used to characterize the quantitative description information of the brightness of the light presented by the lighting atmosphere virtual element through diffuse reflection in the current lighting environment. The reference virtual lighting intensity information can determine the basic brightness level of the reference virtual element in the projected image and the degree of adaptation to the surrounding environment lighting.

[0042] Optionally, the process of determining the reference virtual lighting intensity information of the lighting atmosphere virtual element according to the current light source lighting intensity information, the reference virtual lighting direction information and the diffuse reflection coefficient may refer to the following formula:

[0043] in, Reference virtual lighting intensity information of virtual elements of lighting atmosphere can be represented; Can represent diffuse reflectance; Can represent the current light intensity information of the light source; It can represent the cosine information of the reference virtual lighting direction.

[0044] The reference virtual lighting shadow information of the lighting atmosphere virtual element can be determined by using the current light source perspective depth map, and is used to describe the relevant descriptive information of the shadow situation produced by the lighting atmosphere virtual element in the current lighting environment. The reference virtual lighting shadow information may include but is not limited to information such as the position, shape, size, and depth of the shadow. The shadow effect plays a key role in reflecting the three-dimensional sense of the virtual element and the degree of integration with the environment.

[0045] Specifically, the depth map of the current light source perspective can be analyzed to compare the depth values ​​corresponding to each part of the virtual element with the positional relationship between the surrounding objects and the light source. For example, if part A of the virtual element is shown in the depth map as being farther from the light source than object B, and object B is between the light source and part A of the virtual element, then according to the principle of light occlusion, part A of the virtual element can produce a shadow. The specific shape, size (such as the projection range of the shadow on part A of the virtual element or other receiving surfaces) and depth of the shadow (affected by factors such as light source intensity and distance) can be determined by further analyzing information such as the depth difference.

[0046] The highlight index can be a parameter related to the smoothness of the surface of the lighting atmosphere virtual element, and can be used to control the concentration and diffusion range of the highlight on the surface of the virtual element. The larger the value of the highlight index, the more concentrated and sharper the highlight. Optionally, the highlight effect of the reference virtual element can usually be shaped by simulating the transition effect from smooth to rough surfaces of different materials, working together with the highlight reflection coefficient.

[0047] Specifically, for virtual elements simulating metal materials, if you want to make the surface smooth and shiny, you can set a higher highlight index. At this time, under light, the highlight area will appear very bright, small in area and with clear boundaries, so as to be more in line with the human eye's visual perception of smooth metal highlights; for virtual elements simulating slightly rough wood, the highlight index can be set relatively low, so that the highlight area is relatively more dispersed and the boundaries are more blurred, so that it looks more naturally fit the texture of wood. This application does not impose any restrictions on this.

[0048] The sight direction can indicate the direction in which the observer (which can be a virtual perspective, or an actual human eye or camera viewing the projection) is looking at the virtual element. The reference virtual lighting highlight information of the lighting atmosphere virtual element can be determined by a combination of factors such as the highlight reflection coefficient, highlight index, sight direction, current light source lighting direction information, and preset virtual position information, and is used to describe the relevant descriptive information of the highlight situation generated on the surface of the lighting atmosphere virtual element in the current lighting environment.

[0049] Optionally, the process of determining the reference virtual lighting highlight information of the lighting atmosphere virtual element according to the highlight reflection coefficient, the highlight index, the sight line direction, the lighting direction information of the current light source and the preset virtual position information can be referred to the following formula:

[0050] Among them, I3 can represent the reference virtual lighting highlight information of the lighting atmosphere virtual element; k s Can represent the high light reflection coefficient; I light Can represent the current light intensity information of the light source; It can express the angle between the sight direction and the reflected light direction; It can represent the cosine information of the angle between the sight line direction and the reflected light direction; n can represent the specular index. Specifically, since the reflection angle of light is equal to the incident angle, the reflected light direction can be further calculated according to the law of optical reflection through the current light source illumination direction and the normal direction of the virtual object surface.

[0051] Optionally, the process of determining the lighting atmosphere virtual element image information according to the reference virtual lighting direction information, the reference virtual lighting intensity information, the reference virtual lighting shadow information and the reference virtual lighting highlight information may refer to the following formula:

[0052] Among them, I L It can represent the virtual element image information of the lighting atmosphere; I1 can represent the reference virtual lighting intensity information; I can represent the cosine information of the reference virtual lighting direction; I2 can represent the reference virtual lighting shadow information; and I3 can represent the reference virtual lighting highlight information.

[0053] Referring to the virtual lighting direction information, the difference in lighting received by different surfaces of the virtual elements can be determined, which can create a basis for light and dark contrast. Referring to the virtual lighting intensity information, the overall brightness can be clarified. Referring to the virtual lighting shadow information, the three-dimensional sense and spatial relationship can be created. Referring to the virtual lighting highlight information, the gloss and texture can be reflected. Integrating the above lighting-related information can fully depict the comprehensive information of the appearance of the lighting atmosphere virtual elements under lighting. Graphics rendering and other technologies can be further used to generate corresponding lighting atmosphere virtual element images, so that when they are integrated into the reference projection image, they can perfectly match the environment in terms of lighting effects, achieve realistic reality-enhanced projection effects, and allow the audience to experience the natural fusion of virtual elements and the real lighting environment.

[0054] The humidity atmosphere virtual element image information may be determined based on the current humidity atmosphere state information and the reference projection image information, and may be used to create a visual effect related to humidity in the reference projection image and enhance the feeling of the humidity atmosphere of the environment. The humidity atmosphere virtual element image may mainly reflect the humidity-related atmosphere in the current environment by simulating common water vapor, water droplets, fog and other phenomena in a high humidity environment, or dryness, dust and other visual expressions in a low humidity environment.

[0055] Specifically, in the projection image corresponding to the environment atmosphere with high humidity, the humidity atmosphere virtual element image information can be determined according to the current humidity atmosphere state information and the reference projection image information. At this time, the humidity atmosphere virtual element image information can include image information of virtual water vapor elements and / or virtual fog elements. Optionally, taking the virtual fog element as an example, the humidity atmosphere virtual element image information can include the concentration of the virtual fog element (such as thick fog or thin mist, which can be reflected by the transparency of pixels, color depth, etc.), range (which can involve the size and boundary conditions of the foggy area), dynamic effects (such as the drifting direction and speed of the fog, which can simulate the flow form of the fog in the breeze, etc.), etc., so that the projection image presents a visual effect as if being in a humid environment, such as adding fog elements to a virtual valley scene to create an atmosphere of high humidity and foggy morning in the mountains.

[0056] Specifically, in the projection image corresponding to the low-humidity environment atmosphere, the humidity atmosphere virtual element image information can be determined according to the current humidity atmosphere state information and the reference projection image information. At this time, the humidity atmosphere virtual element image information can include image information of virtual dust elements and / or virtual dry textures. Optionally, taking the virtual dust element as an example, the fine dust particles raised when the wind blows over the ground can be simulated (such as by adding dust particle images with a certain degree of transparency and dynamic effects, the trajectory and speed of the floating can be set); or taking the reference projection image information as a country road as an example, the soil surface of the country road can be added with fine cracks and other texture changes caused by drying to visually enhance the dry atmosphere in the low-humidity environment.

[0057] Optionally, a sound sensor may be used to collect background sounds in the current environment, such as wind and rain, so that the background sounds in the current environment can be obtained from the current environment atmosphere state information, so as to further use the background sounds in the projection display process. For example, if the current environment atmosphere state information indicates that the current environment is a quiet, slightly gloomy rainy night atmosphere, the virtual projection elements may include darker lighting effects and the sound of pattering rain, etc., and this application does not limit this.

[0058] The current environment physical state information may be the relevant attribute information of the objects actually existing in the current environment. Specifically, the geometric shape, material, position and spatial relationship between the objects actually existing in the current environment may be obtained by pre-manual detection or computer vision technology. The current environment physical state information may be used as a basis to determine how the reference virtual elements interact and merge with the objects in the current environment, thereby ensuring that the virtual elements are coordinated with the current environment in terms of spatial layout and functional realization.

[0059] The interactive virtual element image information may be information related to the appearance of the interactive virtual element in the projection scene (including shape, size, material texture, light and shadow effects, etc.) and the presentation status in the projection image, which is determined after comprehensively considering the reference atmosphere effect information, reference shape and size information, reference material texture information, and reference projection position information. The interactive virtual element image information may provide a specific image basis for subsequent operations such as fusing the interactive virtual element with the reference projection image, so that the reference virtual element can be more naturally integrated into the reference projection image, thereby better realizing the projection processing effect of augmented reality.

[0060] It should be understood that determining the interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical state information refers to the process of how to determine the interactive virtual element image information. Among them, step S34, that is, determining the interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical state information, includes the following steps: S341: Acquire first geometric shape information, first size information, first material texture information, and first spatial position information of a first reference object from the current environment object state information; S342: Determine reference shape and size information of the interactive virtual element according to the first geometric shape information and the first size information; S343: Determine reference material texture information of the interactive virtual element according to the first material texture information; S344: determining reference projection position information of the interactive virtual element according to the first spatial position information; S345: performing light and shadow effect processing according to the reference projection image information, the current environment atmosphere state information, the reference shape size information, the reference material texture information and the reference projection position information to obtain reference atmosphere effect information of the interactive virtual element; S346: Determine interactive virtual element image information according to the reference atmosphere effect information, the reference shape size information, the reference material texture information and the reference projection position information.

[0061] The current environment physical object state information may include relevant state information of one or more reference physical objects. Optionally, taking the current environment including a first reference physical object as an example, the current environment physical object state information may include but is not limited to the first geometric shape information, first size information, first material texture information and first spatial position information of the first reference physical object, and this application does not limit this.

[0062] The first geometric shape information may describe in detail the external shape characteristic information of the first reference object in the current environment. Specifically, for regular shaped objects, such as a rectangular table, a cylindrical column, etc., their shapes can be determined by parameters such as length, width, height or radius; for irregular shapes, such as naturally grown trees, unique sculptures, etc., their contours and surface morphology can be accurately depicted with the help of point cloud data, three-dimensional models, etc.

[0063] The first size information can indicate the specific size value of the first reference object, such as the specific length of the length, width, and height (measured in length units such as meters and centimeters), or related size parameters such as radius and diameter (for circular, spherical and other objects). The first size information can determine the size ratio of the reference virtual element that matches it to avoid the situation where the reference virtual element and the real object are spatially incompatible. For example, taking a virtual ornament element placed on a real table as an example, the size of the virtual ornament element needs to match the size of the real table to be more in line with the actual visual experience.

[0064] The first material texture information may indicate the material type and corresponding texture features of the surface of the first reference physical object. The material may be wood, metal, plastic, stone, fabric, etc., and the texture information may include visual information such as the pattern, texture, and roughness or smoothness of the surface. For example, a wooden material may include wood grain texture, and a metal material may include a metallic luster or a specific processing texture. The first material texture information helps to select a matching virtual element material texture, so that the virtual element is visually closer to the real object and enhances the sense of reality of the fusion.

[0065] The first spatial position information can be used to indicate the specific coordinate position of the first reference object in the entire environment space, as well as the relative position relationship between the first reference object and other objects. It can usually be expressed in three-dimensional spatial coordinates (such as x, y, z coordinate values ​​in a preset spatial coordinate system), and can also include the orientation relationship between objects (such as adjacent, up and down, front and back, etc.) and distance information.

[0066] The reference shape and size information of the interactive virtual element can be determined based on the geometric shape information and size information of the first reference object in the current environment, and is used to describe the shape characteristics that the interactive virtual element should have in the projection scene and the relevant information of the corresponding size range. The reference shape and size information can make the reference virtual element adapt to the real object in shape and size, thereby laying the foundation for subsequent fusion and interactive operations.

[0067] Specifically, if the first reference physical object is a rectangular parallelepiped, the reference virtual element can be designed into a rectangular parallelepiped shape that fits or echoes the first reference physical object according to the application scenario, such as designing a virtual rectangular box and placing it on the first reference physical object; if the first reference physical object is an irregularly shaped tree, the reference virtual element can be designed as a virtual vine element that climbs along the shape of the trunk, and the shape of the vine can be determined based on the contour features of the tree such as the direction of the branches.

[0068] The reference material texture information of the interactive virtual element may be determined based on the material texture information of the first reference object in the current environment, and is used to describe the type of material that the interactive virtual element should use in the projection scene and the related information of the corresponding texture features. The reference material texture information can make the reference virtual element correspond to the real object in terms of material texture, so as to further enhance the realism and integration of the virtual element in the projection image.

[0069] Specifically, if the first reference object is made of wood, the interactive virtual element can give priority to choosing wood or a material with a similar visual effect, so that the reference virtual element looks like it is made of a similar material to the object. After determining the material, a texture mapping operation is performed on the texture features corresponding to the material, such as applying appropriate texture patterns, lines, etc. to the surface of the virtual element, so that it presents a surface texture similar to that of a real object. Exemplarily, for the first reference object made of metal, if the reference virtual element adopts a metal material, the corresponding metallic gloss texture and processing lines can be accurately mapped to the surface of the reference virtual element, so that its optical performance such as reflection and scattering under light is more consistent with that of a real metal object. Through the matching of the above-mentioned material textures, the reference virtual element can be better integrated into the real environment to enhance the overall projection effect.

[0070] The reference projection position information of the interactive virtual element can be determined based on the spatial position information of the first reference object in the current environment, and is used to describe the specific position where the interactive virtual element should be placed in the projection scene, as well as the corresponding spatial coordinates, posture and other related information. The reference projection position information can better ensure that the virtual element is in line with the real environment in terms of spatial layout, and is more in line with people's conventional understanding of the position relationship of objects in space, so that the interactive function between the virtual element and the real object can be realized in the subsequent steps.

[0071] Specifically, the spatial position of the first reference object can usually be used as a basis to further determine the projection position of the reference virtual element. For example, taking the first reference object as a flower pot in the corner of a wall in a real environment, if you want to add a virtual flower element in the flower pot, you can set the position of the flower to the corresponding spatial position inside the flower pot. Optionally, appropriate fine-tuning can be performed based on factors such as the overall layout of the projection scene, visual aesthetics, and interaction requirements with other virtual or real elements. For example, in order to avoid occlusion conflicts between virtual elements and other objects in the projected image, or to create a better sense of spatial hierarchy, the virtual element may be slightly translated or rotated by a certain angle to further determine its final projection position.

[0072] The reference atmosphere effect information of the interactive virtual element can be the information obtained by integrating the reference projection image information, the current environment atmosphere state information, the reference shape size information, the reference material texture information and the reference projection position information, and processed by the light and shadow effect, and is used to describe the lighting, shadow, highlight and overall light and shadow atmosphere effects presented by the interactive virtual element in the environment atmosphere of the current projection scene. The reference atmosphere effect information can make the reference virtual element more compatible with the reference projection image and the real environment in terms of light and shadow performance, thereby enhancing the realism and immersion of the virtual element.

[0073] Specifically, the shadow projection situation of the virtual element (including the shape, size, position, etc. of the shadow) can be calculated based on the position and direction of the light source and the shape, position and other information of the virtual element, so that it matches the shadow performance of objects in the real environment; at the same time, the highlight position, intensity and distribution range of the surface of the virtual element can be determined according to the highlight reflection characteristics of the material (refer to the material texture information) and the observer's line of sight and other factors, so as to create a gloss effect consistent with the texture of its material. Further comprehensive processing of the above-mentioned light and shadow effects can make the virtual element present a realistic light and shadow atmosphere in the projection scene, and then naturally blend with the surrounding environment.

[0074] The reference shape size information can determine the basic geometric appearance of the reference virtual element, so that it has a clear size and shape in space; the reference material texture information can give the reference virtual element surface texture and visual characteristics, making it look more real; the reference projection position information can standardize the placement of the reference virtual element in the projection scene, making it more in line with the spatial layout logic; the reference atmosphere effect information can make the virtual element more compatible with the current environment in terms of environmental atmosphere (such as light and shadow performance such as lighting), thereby creating a realistic visual effect. By integrating the above information, the appearance of the interactive virtual element in the projection scene can be fully portrayed, and further through graphics rendering and other technologies, the corresponding interactive virtual element image can be generated, so that it can achieve a better reality enhancement effect in terms of appearance and interactivity with the environment, thereby enhancing the realism and immersion of the entire projection scene.

[0075] S40: Performing reality enhancement processing on the reference virtual element image information and the reference projection image information to obtain target projection image information.

[0076] Among them, the target projection image information can be a more immersive projection image obtained after the reference virtual element image and the reference projection image are subjected to reality enhancement processing. Specifically, the target projection image information can include the position information after the reference virtual element and the reference projection image are accurately aligned in space, so as to ensure that the placement of the reference virtual element in the reference projection image is more in line with the logic and visual aesthetics of the real space; it can include the pixel fusion effect processed by transparency fusion and other methods, so that the reference virtual element and the reference projection image have a natural transition; and it can include the light and shadow effect after comprehensive consideration of the overall light and shadow coordination, so that the reference virtual element and the scenery in the reference projection image can be consistent under the lighting. Through the reality enhancement fusion processing in the above aspects, the projection effect of mutual fusion and mutual complementation between the reference virtual element and the reference projection image can be more completely presented, and finally the target projection image information can be obtained.

[0077] It should be understood that performing reality enhancement processing on the reference virtual element image information and the reference projection image information to obtain the target projection image information refers to the process of how to obtain the target projection image information. Among them, step S40, that is, performing reality enhancement processing on the reference virtual element image information and the reference projection image information to obtain the target projection image information, includes the following steps: S41: acquiring a first feature point from the reference virtual element image information, and acquiring a second feature point corresponding to the first feature point from the reference projection image information; S42: performing spatial registration processing according to the first feature point and the second feature point to obtain spatial information of the target projection image; S43: performing transparency fusion processing on the reference virtual element image information and the reference projection image information to obtain transparency information of the target projection image; S44: Determine target projection image information according to the spatial information of the target projection image and the transparency information of the target projection image.

[0078] The first feature point may be a representative key point extracted from the reference virtual element image information that can help with image matching and subsequent processing. The first feature point may have unique visual features in the reference virtual element image, such as a corner point of an object in the reference virtual element image, an edge turning point, a point with obvious texture changes, etc. The first feature point may be used as a feature identifier to find the corresponding relationship between the reference virtual element image and the reference projection image, thereby achieving more accurate fusion processing.

[0079] Specifically, the Harris corner detection algorithm can be used, which can be based on the grayscale changes in the local area of ​​the image and calculate the eigenvalues ​​of the autocorrelation matrix to determine the corners where the grayscale changes significantly in multiple directions. The Scale-Invariant Feature Transform (SIFT) algorithm can be used to detect feature points in different scale spaces, and a corresponding feature descriptor can be generated for each feature point to describe the image features of the local area around the feature point, thereby facilitating subsequent matching operations.

[0080] The second feature point may be a key point extracted from the reference projection image information and having a matching relationship with the first feature point. The second feature point may be a representative point in the reference projection image. Specifically, a second feature point corresponding to the first feature point may be found through a specific matching algorithm, so that the transformation relationship between the reference projection image and the reference virtual element image may be determined based on the spatial relationship between the second feature point and the first feature point, and then spatial registration and other operations may be implemented to ensure that the virtual element is accurately integrated into the appropriate position in the projection image.

[0081] Specifically, the correspondence can be determined by calculating the similarity between the feature descriptors of the two feature points (such as the feature descriptors generated by the aforementioned SIFT algorithm). Optionally, the similarity between the two feature points can be determined by calculating the Euclidean distance or the like. For example, the Euclidean distance between the feature descriptor of the first feature point and the feature descriptors of all the second feature points in the reference projection image can be calculated, and the feature point with the smallest distance and meeting the preset threshold condition can be used as the corresponding second feature point.

[0082] Based on the aforementioned first feature point and second feature point, the spatial transformation relationship between the reference virtual element image and the reference projection image is determined. After spatial registration processing is performed according to the spatial transformation relationship, the reference virtual element can be accurately placed at the spatial position corresponding to the reference projection image, so that the two can be aligned in space and the spatial information of the target projection image can be obtained, which can ensure that the virtual element is more in line with the realistic spatial logic and visual aesthetics when integrated into the projection image.

[0083] Furthermore, by mixing the reference virtual element image information with the reference projection image information according to a preset transparency rule, transparency information of the target projection image that is used to describe the transparency information of each pixel in the target projection image can be obtained, so that the reference virtual element can present different degrees of transparency when integrated into the reference projection image, thereby making the fusion transition more natural and enhancing the sense of fusion between the reference virtual element and the reference projection image.

[0084] Specifically, the above transparency fusion processing process can be performed based on the transparency channel (alpha channel) fusion method. For example, taking the reference virtual element image with an alpha channel as an example, the alpha value of each pixel in the reference virtual element image can represent the transparency of the reference virtual element at the pixel position, where the value range is 0 to 1, 0 can represent completely transparent, and 1 can represent completely opaque.

[0085]

[0086] Among them, R aim It can represent the transparency information on the red channel in the transparency information of the target projection image; R m It can be used to represent the color value of the pixel in the reference projection image on the R channel; It can be used to represent the alpha value of the pixel corresponding to the reference virtual element image in the reference virtual element image information; It can be used to represent the color value of the pixel in the reference virtual element image on the R channel; It can represent the transparency information on the green channel in the transparency information of the target projection image; It can be used to represent the color value of the pixel in the reference projection image on the G channel; It can be used to represent the color value of the pixel in the reference virtual element image on the G channel; It can represent the transparency information on the blue channel in the transparency information of the target projection image; It can be used to represent the color value of the pixel in the reference projection image on the B channel; It can be used to represent the color value of the pixel in the reference virtual element image on the B channel.

[0087] By calculating pixel by pixel in the above manner, the overall transparency information of the target projection image can be obtained, so that the reference virtual elements can be naturally integrated into the reference projection image. For example, some translucent effects in the reference virtual elements (such as virtual translucent smoke, virtual translucent light and shadow, etc.) can be more reasonably displayed in the reference projection image through transparency fusion processing.

[0088] S50: Projecting and displaying the target projection image information in the current projection space and the current environment state.

[0089] After obtaining the above-mentioned target projection image information, at this time, according to the current projection space and the current environmental conditions, with the help of projection equipment (such as a high-definition projector and / or a holographic projector, etc.), the target projection image information can be more accurately and appropriately projected into the current projection space.

[0090] Specifically, the projection device can present the target projection image that integrates the reference virtual elements completely and clearly in the current projection space based on the spatial layout, pixel color, light and shadow effects, transparency and other data contained in the target projection image information, under the influence of the current environmental conditions such as lighting, temperature, humidity and existing real objects. This allows the audience in the space to more intuitively see the effect of the perfect integration of virtual elements and the real environment, and to better immerse themselves in the unique visual experience after the reality augmented processing.

[0091] It can be understood that the target projection image information can be projection image information that is more in line with the current projection space and the current environmental state. When the current projection space changes, such as when the projection device moves to another projection space, the more appropriate projection display image information can be re-determined based on the changed projection space information; or when the current environmental state changes, such as when the current environment changes from sunny to rainy, and the light intensity or humidity changes, the more appropriate projection display image information can be re-determined based on the changed environmental state information. The present application does not impose any restrictions on this.

[0092] It can be seen that in the above scheme, by acquiring the current projection space information, the current environmental state information and the optically related information to be projected, the reference projection image information can be determined according to the current projection space information and the optically related information to be projected, and the reference virtual element image information can be determined according to the current environmental state information and the reference projection image information, so that the reference virtual element image information and the reference projection image information can be subjected to reality enhancement processing to obtain the target projection image information, and then the target projection image information can be projected and displayed in the current projection space and the current environmental state, which can improve the effectiveness of the projection display and is conducive to achieving the purpose of more effective, richer and more interesting projection display.

[0093] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0094] In one embodiment, a projection image processing device based on reality enhancement is provided, and the projection image processing device based on reality enhancement corresponds to the projection image processing method based on reality enhancement in the above embodiment. Figure 2 As shown, the projection image processing device based on augmented reality includes an acquisition module 101, a determination module 102 and a processing module 103. Each functional module is described in detail as follows: An acquisition module 101 is used to acquire current projection space information, current environment state information and optical related information to be projected; A first determining module 102 is used to determine reference projection image information according to the current projection space information and the optical related information to be projected; A second determining module 103, configured to determine reference virtual element image information according to the current environment state information and the reference projection image information; A processing module 104 is used to perform reality enhancement processing on the reference virtual element image information and the reference projection image information to obtain target projection image information; The display module 105 is used to project and display the target projection image information in the current projection space and the current environment state.

[0095] In one embodiment, the second determining module 103 is used to determine the reference virtual element image information according to the current environment state information and the reference projection image information, specifically for: Acquire current environment atmosphere state information from the current environment state information, and acquire current environment physical state information from the current environment state information; Determining lighting atmosphere virtual element image information according to current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information; Determining humidity atmosphere virtual element image information according to current humidity atmosphere state information in the current environmental atmosphere state information and the reference projection image information; Determining interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical object state information; Reference virtual element image information is determined according to the lighting atmosphere virtual element and the interactive virtual element.

[0096] In one embodiment, the second determining module 103 is used to determine the lighting atmosphere virtual element image information according to the current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information, specifically for: Acquire the current light source illumination intensity information, the current light source illumination direction information and the current light source viewing angle depth map from the current light atmosphere state information; Determining reference virtual lighting direction information of a lighting atmosphere virtual element according to the current light source lighting direction information and preset virtual position information in the reference projection image information; Determining reference virtual lighting intensity information of a lighting atmosphere virtual element according to the current light source lighting intensity information, the reference virtual lighting direction information and the diffuse reflection coefficient; Determining reference virtual lighting shadow information of the lighting atmosphere virtual element according to the current light source viewing angle depth map; Determine reference virtual lighting highlight information of the lighting atmosphere virtual element according to the highlight reflection coefficient, the highlight index, the sight direction, the lighting direction information of the current light source and the preset virtual position information; The lighting atmosphere virtual element image information is determined according to the reference virtual lighting direction information, the reference virtual lighting intensity information, the reference virtual lighting shadow information and the reference virtual lighting highlight information.

[0097] In one embodiment, the second determining module 103 is used to determine the interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical state information, specifically for: Acquire first geometric shape information, first size information, first material texture information and first spatial position information of a first reference object from the current environment object state information; Determining reference shape and size information of the interactive virtual element according to the first geometric shape information and the first size information; Determining reference material texture information of the interactive virtual element according to the first material texture information; Determining reference projection position information of the interactive virtual element according to the first spatial position information; Performing light and shadow effect processing according to the reference projection image information, the current environment atmosphere state information, the reference shape size information, the reference material texture information and the reference projection position information to obtain reference atmosphere effect information of the interactive virtual element; The interactive virtual element image information is determined according to the reference atmosphere effect information, the reference shape size information, the reference material texture information and the reference projection position information.

[0098] In one embodiment, the processing module 104 is used to perform reality augmentation processing on the reference virtual element image information and the reference projection image information to obtain target projection image information, specifically for: Acquire a first feature point from the reference virtual element image information, and acquire a second feature point corresponding to the first feature point from the reference projection image information; Performing spatial registration processing according to the first feature point and the second feature point to obtain spatial information of the target projection image; Performing transparency fusion processing on the reference virtual element image information and the reference projection image information to obtain transparency information of the target projection image; The target projection image information is determined according to the spatial information of the target projection image and the transparency information of the target projection image.

[0099] The present invention provides a projection image processing device based on reality augmentation. By acquiring current projection space information, current environment state information and optical related information to be projected, reference projection image information can be determined according to the current projection space information and the optical related information to be projected, and reference virtual element image information can be determined according to the current environment state information and the reference projection image information, so that the reference virtual element image information and the reference projection image information can be subjected to reality augmentation processing to obtain target projection image information, and then the target projection image information can be projected and displayed in the current projection space and the current environment state, which can improve the effectiveness of the projection display and is conducive to achieving the purpose of more effective, richer and more interesting projection display.

[0100] For the specific definition of the projection image processing device based on reality enhancement, please refer to the definition of the projection image processing method based on reality enhancement above, which will not be repeated here. Each module in the above-mentioned projection image processing device based on reality enhancement can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0101] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a projection image processing method based on augmented reality.

[0102] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes a function or step on the client side of a projection image processing method based on augmented reality.

[0103] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtain current projection space information, current environment status information and optical related information to be projected; Determine reference projection image information according to the current projection space information and the optical related information to be projected; Determining reference virtual element image information according to the current environment state information and the reference projection image information; Performing reality augmentation processing on the reference virtual element image information and the reference projection image information to obtain target projection image information; In the current projection space and the current environment state, the target projection image information is projected and displayed.

[0104] The present invention provides a computer device, which can determine reference projection image information according to the current projection space information and the optical related information to be projected by acquiring current projection space information, current environment state information and the optical related information to be projected, and can determine reference virtual element image information according to the current environment state information and the reference projection image information, so that the reference virtual element image information and the reference projection image information can be subjected to reality enhancement processing to obtain target projection image information, and then the target projection image information can be projected and displayed in the current projection space and the current environment state, which can improve the effectiveness of the projection display and is conducive to achieving the purpose of more effective, richer and more interesting projection display.

[0105] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtain current projection space information, current environment status information and optical related information to be projected; Determine reference projection image information according to the current projection space information and the optical related information to be projected; Determining reference virtual element image information according to the current environment state information and the reference projection image information; Performing reality augmentation processing on the reference virtual element image information and the reference projection image information to obtain target projection image information; In the current projection space and the current environment state, the target projection image information is projected and displayed.

[0106] The present invention provides a computer-readable storage medium, which can determine reference projection image information according to the current projection space information and the optically related information to be projected by acquiring current projection space information, current environmental state information and the optically related information to be projected, and can determine reference virtual element image information according to the current environmental state information and the reference projection image information, so that the reference virtual element image information and the reference projection image information can be subjected to reality enhancement processing to obtain target projection image information, and then the target projection image information can be projected and displayed in the current projection space and the current environmental state, which can improve the effectiveness of the projection display and is conducive to achieving the purpose of more effective, richer and more interesting projection display.

[0107] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0108] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0109] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0110] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A projection image processing method based on reality augmentation, characterized in that: The method comprises: Obtain current projection space information, current environment status information and optical related information to be projected; Determine reference projection image information according to the current projection space information and the optical related information to be projected; Determining reference virtual element image information according to the current environment state information and the reference projection image information; Performing reality augmentation processing on the reference virtual element image information and the reference projection image information to obtain target projection image information; In the current projection space and the current environment state, the target projection image information is projected and displayed.

2. The projection image processing method based on reality augmentation according to claim 1, characterized in that: The determining, according to the current environment state information and the reference projection image information, reference virtual element image information includes: Acquire current environment atmosphere state information from the current environment state information, and acquire current environment physical state information from the current environment state information; Determining lighting atmosphere virtual element image information according to current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information; Determining humidity atmosphere virtual element image information according to current humidity atmosphere state information in the current environmental atmosphere state information and the reference projection image information; Determining interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical object state information; Reference virtual element image information is determined according to the lighting atmosphere virtual element and the interactive virtual element.

3. The projection image processing method based on reality augmentation according to claim 2, characterized in that: The determining, according to the current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information, the lighting atmosphere virtual element image information comprises: Acquire the current light source illumination intensity information, the current light source illumination direction information and the current light source viewing angle depth map from the current light atmosphere state information; Determining reference virtual lighting direction information of a lighting atmosphere virtual element according to the current light source lighting direction information and preset virtual position information in the reference projection image information; Determining reference virtual lighting intensity information of a lighting atmosphere virtual element according to the current light source lighting intensity information, the reference virtual lighting direction information and the diffuse reflection coefficient; Determining reference virtual lighting shadow information of the lighting atmosphere virtual element according to the current light source viewing angle depth map; Determine reference virtual lighting highlight information of the lighting atmosphere virtual element according to the highlight reflection coefficient, the highlight index, the sight direction, the lighting direction information of the current light source and the preset virtual position information; The lighting atmosphere virtual element image information is determined according to the reference virtual lighting direction information, the reference virtual lighting intensity information, the reference virtual lighting shadow information and the reference virtual lighting highlight information.

4. The projection image processing method based on reality augmentation according to claim 3, characterized in that: The step of determining the interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical state information includes: Acquire first geometric shape information, first size information, first material texture information and first spatial position information of a first reference object from the current environment object state information; Determining reference shape and size information of the interactive virtual element according to the first geometric shape information and the first size information; Determining reference material texture information of the interactive virtual element according to the first material texture information; Determining reference projection position information of the interactive virtual element according to the first spatial position information; Performing light and shadow effect processing according to the reference projection image information, the current environment atmosphere state information, the reference shape size information, the reference material texture information and the reference projection position information to obtain reference atmosphere effect information of the interactive virtual element; The interactive virtual element image information is determined according to the reference atmosphere effect information, the reference shape size information, the reference material texture information and the reference projection position information.

5. The projection image processing method based on reality augmentation according to any one of claims 1 to 4, characterized in that: The step of performing reality augmentation processing on the reference virtual element image information and the reference projection image information to obtain target projection image information includes: Acquire a first feature point from the reference virtual element image information, and acquire a second feature point corresponding to the first feature point from the reference projection image information; Performing spatial registration processing according to the first feature point and the second feature point to obtain spatial information of the target projection image; Performing transparency fusion processing on the reference virtual element image information and the reference projection image information to obtain transparency information of the target projection image; The target projection image information is determined according to the spatial information of the target projection image and the transparency information of the target projection image.

6. A projection image processing device based on reality augmentation, characterized in that: The projection image processing device based on reality augmentation includes: An acquisition module is used to acquire current projection space information, current environment status information and optical related information to be projected; A first determination module, configured to determine reference projection image information according to the current projection space information and the optical related information to be projected; A second determining module, configured to determine reference virtual element image information according to the current environment state information and the reference projection image information; A processing module, used for performing reality enhancement processing on the reference virtual element image information and the reference projection image information to obtain target projection image information; The display module is used to project and display the target projection image information in the current projection space and the current environment state.

7. The projection image processing device based on augmented reality according to claim 6, characterized in that: The second determination module is used to determine the reference virtual element image information according to the current environment state information and the reference projection image information, and is specifically used to: Acquire current environment atmosphere state information from the current environment state information, and acquire current environment physical state information from the current environment state information; Determining lighting atmosphere virtual element image information according to current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information; Determining humidity atmosphere virtual element image information according to current humidity atmosphere state information in the current environmental atmosphere state information and the reference projection image information; Determining interactive virtual element image information according to the reference projection image information, the current environment atmosphere state information and the current environment physical object state information; Reference virtual element image information is determined according to the lighting atmosphere virtual element and the interactive virtual element.

8. The projection image processing device based on augmented reality according to claim 7, characterized in that: The second determination module is used to determine the lighting atmosphere virtual element image information according to the current lighting atmosphere state information in the current environment atmosphere state information and the reference projection image information, and is specifically used to: Acquire the current light source illumination intensity information, the current light source illumination direction information and the current light source viewing angle depth map from the current light atmosphere state information; Determining reference virtual lighting direction information of a lighting atmosphere virtual element according to the current light source lighting direction information and preset virtual position information in the reference projection image information; Determining reference virtual lighting intensity information of a lighting atmosphere virtual element according to the current light source lighting intensity information, the reference virtual lighting direction information and the diffuse reflection coefficient; Determining reference virtual lighting shadow information of the lighting atmosphere virtual element according to the current light source viewing angle depth map; Determine reference virtual lighting highlight information of the lighting atmosphere virtual element according to the highlight reflection coefficient, the highlight index, the sight direction, the lighting direction information of the current light source and the preset virtual position information; The lighting atmosphere virtual element image information is determined according to the reference virtual lighting direction information, the reference virtual lighting intensity information, the reference virtual lighting shadow information and the reference virtual lighting highlight information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the projection image processing method based on augmented reality as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the projection image processing method based on augmented reality as described in any one of claims 1 to 5 is implemented.

Images