Fusion display method and device, electronic equipment and storage medium

Abstract

The invention provides a fusion display method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing, and the method comprises the steps: determining a real scene element and a task guide element based on a real scene image shot by a camera in an augmented reality module; placing a 3D display image generated by using the real scene element in a first display layer of a user display interface in the augmented reality module; and placing a 2D display image generated by using the task guide element in a second display layer of the user display interface in the augmented reality module, wherein the second display layer is superposed on the first display layer. According to the fusion display method and device, the electronic equipment and the storage medium provided by the invention, the elements which need to be carefully concerned and read by a user are designed to be 2D, the time delay of the motion trail of the coordinate system in the virtual world in the augmented reality module can be effectively reduced, and the equipment cost input and power consumption are reduced on the basis of ensuring the availability of the augmented reality module.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a fusion display method, device, electronic device and storage medium. Background Art

[0002] In order to improve the immersion and realism of the virtual world and avoid motion sickness and discomfort in users, ideally, the coordinate system of the virtual world should be completely synchronized with the coordinate system of the user's physical space, and the angle of sight and the position of the viewpoint in the two need to be consistent.

[0003] However, in actual applications of the virtual world, the movement trajectory of the coordinate system in the virtual world is likely to be tens of milliseconds or even close to a hundred milliseconds slower than the movement trajectory of the user's body. The above delay will not have a significant impact on the user if the user's physical position remains stable, but if the user quickly rotates the viewing angle, based on the average human head rotation of 200 degrees per second, a 20-millisecond delay may result in a deviation of up to 4 degrees (equivalent to the angle between the two sides of a paper cup held at arm's length).

[0004] In order to solve the technical problem that the motion trajectory of the coordinate system in the virtual world is delayed compared to the motion trajectory of the user's body in the related art, in addition to improving the head tracking accuracy, the time lag of the motion trajectory of the coordinate system in the virtual world can also be reduced by increasing the refresh rate of the head display. However, the above method significantly increases the equipment investment cost and equipment power consumption. Summary of the invention

[0005] The present invention provides a fusion display method, device, electronic device and storage medium, which are used to solve the defects in the prior art that the equipment cost required to reduce the time lag of the motion trajectory of the coordinate system in the virtual world is high and the power consumption of the equipment is increased, so as to reduce the time lag of the motion trajectory of the coordinate system in the virtual world with lower equipment cost investment and lower equipment power consumption.

[0006] The present invention provides a fusion display method, comprising: obtaining a real scene image captured by a camera in an augmented reality module, wherein the augmented reality module comprises a camera and a user display interface; determining real scene elements according to location information of objects in the real scene image that changes with the location of a target user, and determining task guidance elements according to intelligence information contained in the real scene image, wherein the target user is a user of the augmented reality module; placing a 3D display image generated using the real scene elements in a first display layer of the user display interface in the augmented reality module; placing a 2D display image generated using the task guidance elements in a second display layer of the user display interface in the augmented reality module; wherein the second display layer is superimposed on the first display layer.

[0007] According to a fusion display method provided by the present invention, the 2D display image generated by using the task guidance element is placed in the second display layer of the user display interface in the augmented reality module, including: visualizing the task guidance elements belonging to the target category to generate the 2D display image; setting the projection mode of the 2D display image to orthogonal projection, and displaying the 2D display image in the target orthogonal projection area in the second display layer; wherein the target category is any one of the categories possessed by the task guidance element; the target orthogonal projection area is the area at the edge of the second display layer, and the number of the target orthogonal projection areas is the same as the number of the target categories.

[0008] According to a fusion display method provided by the present invention, after setting the projection mode of the 2D display image to orthogonal projection and displaying the 2D display image in a target orthogonal projection area in the second display layer, the method further includes: receiving and responding to a first operation instruction to fold or unfold the target orthogonal projection area; receiving and responding to a second operation instruction to update the 2D display image; wherein the first operation instruction and the second operation instruction are obtained by recognizing the voice content emitted by the target user and / or by recognizing the gestures of the target user.

[0009] According to a fusion display method provided by the present invention, the 3D display image generated by using the real scene elements is placed in the first display layer of the user display interface in the augmented reality module, including: generating the 3D display image according to the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system; setting the projection mode of the 3D display image to perspective projection, and displaying the 3D display image on the first display layer.

[0010] According to a fusion display method provided by the present invention, the 3D display image is generated according to the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system, including: drawing virtual objects on the first display layer according to the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system; and synthesizing the virtual objects with the real scene image to obtain the 3D display image.

[0011] According to a fusion display method provided by the present invention, the task guidance element includes one or more of a text box, a target plane map, navigation guidance information, and a video stream; wherein the target plane map includes a plane map centered on the target user, and / or a three-dimensional map centered on the target object sighted by the augmented reality module.

[0012] According to a fusion display method provided by the present invention, the real scene elements include one or more of marking points, member positions, assembly points, and positions of target objects aimed at by the augmented reality module; wherein the marking points have different marking graphics according to different marking sources.

[0013] The present invention also provides a fusion display device, comprising: an image acquisition module, used to acquire a real scene image taken by a camera in an augmented reality module, the augmented reality module comprising a camera and a user display interface; a scene analysis module, used to determine real scene elements according to the location information of objects in the real scene image that changes with the location of a target user, and to determine task guidance elements according to the intelligence information contained in the real scene image, the target user being a user of the augmented reality module; an enhanced display module, used to place a 3D display image generated using the real scene elements in a first display layer of the user display interface in the augmented reality module; a fixed display module, used to place a 2D display image generated using the task guidance elements in a second display layer of the user display interface in the augmented reality module; wherein the second display layer is superimposed on the first display layer.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned fusion display methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the fusion display method described in any one of the above is implemented.

[0016] The fusion display method, device, electronic device and storage medium provided by the present invention classify the content to be displayed in the real scene image captured by the augmented reality module into real scene elements and task guidance elements, and use 3D display for the real scene elements that will change with the position of the target user, and use 2D display for the task guidance elements that will not change with the position of the target user and are fixedly covered on the target area of ​​the 3D display image, so that the elements that require the target user to pay close attention to and read are designed as 2D and fixed in front of the eyes of the target user in the form of head-up display. Since the movement of the head does not affect the position of these elements, the refresh rate of the near-eye display can be higher than 60Hz, which can effectively reduce the time lag of the motion trajectory of the coordinate system in the virtual world in the augmented reality module, and can effectively reduce the dizziness and discomfort caused by the time lag of the motion trajectory of the coordinate system in the virtual world in the augmented reality module, thereby reducing the equipment cost investment and power consumption on the basis of ensuring the availability of the augmented reality module. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the fusion display method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the structure of the fusion display device provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0023] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present invention, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0024] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0025] Figure 1 Schematic diagram of the fusion display method provided by the present invention. Figure 1As shown, the fusion display method provided by the embodiment of the present invention includes: step 101, obtaining a real scene image captured by a camera in an augmented reality module, and the augmented reality module includes a camera and a user display interface.

[0026] It should be noted that the executor of the fusion display method is a fusion display device, which can be integrated on a control chip in the augmented reality module.

[0027] In the embodiment of the present invention, the user of the augmented reality module is determined as the target user, and the augmented reality module in the embodiment of the present invention can be used to apply virtual information to the real world. The camera in the augmented reality module can be used to collect real scene images of the real scene where the target user is located, and the user display interface in the augmented reality module can display virtual scene images for the target user to view.

[0028] The application scenario of the fusion display method provided by the embodiment of the present invention is: in the process of the target user applying virtual information to the real world through the augmented reality module, the environmental information that changes with the target user's position is provided to the target user for perception, and the information that requires the target user to pay close attention to or read is projected onto the user display interface in the augmented reality module in the form of a head-up display (HUD) for the user to view.

[0029] Step 102: determine the real scene elements according to the location information of the features in the real scene image that changes with the location of the target user, and determine the task guidance elements according to the intelligence information contained in the real scene image. The target user is the user of the augmented reality module.

[0030] Specifically, in step 102, the fusion display device needs to classify the information elements to be displayed in the real scene image captured by the camera in the augmented reality module according to the actual task requirements of the target user. One type is the real scene element. When the position of the target user changes in the real world, this type of element will change with the scene changes perceived by the target user through the field of view of the augmented reality module. The other type is the task guidance element, which is the content that requires the target user to continue to pay attention to, and will not change with the scene changes in the field of view.

[0031] The embodiment of the present invention does not specifically limit the method of obtaining the task guidance element.

[0032] For example, the direction indicated by the combat mission can be extracted from the real scene image captured by the camera in the augmented reality module in combination with the actual combat mission.

[0033] For example, the combat plan or battlefield situation can be extracted from the intelligence obtained from the interaction with other devices in combination with actual combat missions.

[0034] Step 103: placing the 3D display image generated by using the real scene elements in the first display layer of the user display interface in the augmented reality module.

[0035] Specifically, in step 103, after the fusion display device aligns the landmark feature through the augmented reality module, it uses the landmark feature as a real scene element, obtains the position information of the landmark feature in its field of view in the earth coordinate system, and then combines the position and posture information of the target user relative to each real scene element to solve the projection transformation model of the virtual object and the real scene. Finally, the virtual information corresponding to the real scene element is superimposed on the real scene image taken by the camera in the augmented reality module. The generated 3D display image is displayed in the first display layer at the bottom layer of the user display interface using an augmented reality or mixed reality display solution.

[0036] Among them, the size of the 3D display image is the same as the size of the user display interface in the augmented reality module. During the full-screen display process, the 3D display image will continuously change the 3D rendering content it contains as the position of the human body changes.

[0037] Step 104: placing the 2D display image generated by using the task guidance element in the second display layer of the user display interface in the augmented reality module.

[0038] The second display layer is superimposed on the first display layer.

[0039] Specifically, in step 104, the fusion display device can visualize the task guidance elements determined from the real scene image or the intelligence information contained in other devices to obtain a 2D display image, and place the obtained 2D display image in a second display layer superimposed on the first display layer in the user display interface for fixed display relative to the display.

[0040] The embodiment of the present invention classifies the content that needs to be displayed in the real scene image captured by the augmented reality module into real scene elements and task guidance elements. The real scene elements that will change with the position of the target user are displayed in 3D, and the task guidance elements that will not change with the position of the target user are displayed in 2D and fixedly covered on the target area of ​​the 3D display image. The elements that require the target user to pay close attention to and read are designed as 2D and fixed in front of the eyes of the target user in the form of a head-up display. Since the movement of the head does not affect the position of these elements, the refresh rate of the near-eye display can be higher than 60Hz, which can effectively reduce the time lag of the motion trajectory of the coordinate system in the virtual world in the augmented reality module, and can effectively reduce the dizziness and discomfort caused by the time lag of the motion trajectory of the coordinate system in the virtual world in the augmented reality module, thereby reducing the equipment cost investment and power consumption on the basis of ensuring the availability of the augmented reality module.

[0041] Based on any of the above embodiments, a 2D display image generated by using the task guidance element is placed in the second display layer of the user display interface in the augmented reality module, including: visualizing the task guidance elements belonging to the target category to generate a 2D display image.

[0042] The target category is any one of the categories possessed by the task guidance element.

[0043] It should be noted that the task guidance elements can be summarized into multiple categories according to the information dimension, and the target category refers to one or more categories with higher visualization priority.

[0044] Specifically, the fusion display device extracts the task guidance elements belonging to the target category from the task guidance elements that require the target user to pay close attention to and read, and performs visualization processing on the relevant information to obtain a 2D display image that displays each pair of task guidance elements of the target category.

[0045] The 2D display image includes but is not limited to an interface UI, text, sensor video, data and other data, which is not specifically limited in the embodiment of the present invention.

[0046] The projection mode of the 2D display image is set to orthogonal projection, and the 2D display image is displayed in a target orthogonal projection area in the second display layer.

[0047] The target orthogonal projection area is an area at the edge of the second display layer, and the number of target orthogonal projection areas is the same as the number of target categories. Specifically, the fusion display device sets the projection mode of the 2D display image to orthographic projection, only considering the X and Y coordinates of all points in the three-dimensional space, and mapping the 2D display image that displays the task guidance elements of each pair of target categories in a perspective-free manner into the target orthogonal projection area for rendering.

[0048] The target orthogonal projection area refers to the edge plane area that covers the two-dimensional plane where the second display layer is located and does not block the central field of view. Regardless of how the target user's head rotates, the relative position relationship between the target orthogonal projection area and the target user's eyes remains unchanged. The embodiment of the present invention does not specifically limit the division of the target orthogonal projection area.

[0049] Exemplarily, the target orthogonal projection area may include a sub-area at the corner point position divided from the second display layer by taking the corner point of the second display layer as a new vertex.

[0050] Exemplarily, the target orthogonal projection area may include a sub-area at the edge divided from the second display layer by taking the edge of the second display layer as the new edge.

[0051] The embodiment of the present invention projects the visual content of the task guidance element by orthogonal projection to obtain a corresponding 2D display image, and places the 2D display image on the target orthogonal projection area. The layer for 2D display can be located at the top layer and fixed relative to the user display interface to avoid dizziness, and the user can browse the whole layer without turning the head.

[0052] Based on any of the above embodiments, after setting the projection mode of the 2D display image to orthogonal projection so that the 2D display image is displayed in the target orthogonal projection area, it also includes: receiving and responding to a first operation instruction to fold or unfold the target orthogonal projection area.

[0053] Specifically, after projecting the complete 2D layer, the fusion display device can also receive a first operation instruction obtained by recognizing the voice content and / or gesture issued by the target user through the sensor, and execute corresponding operations in turn according to the control instructions contained in the first operation instruction to complete the folding or unfolding of the target orthogonal projection area.

[0054] The first operation instruction is used to adjust the target orthogonal projection area. The target user can select a number of virtual controls in the target orthogonal projection area to complete the size adjustment of the target orthogonal projection area, and the adjustment method includes but is not limited to a zoom operation, a folding operation, and an unfolding operation, which is not specifically limited in the embodiment of the present invention.

[0055] Optionally, the first operation instruction may be generated through touch input or key input of a physical control, as well as voice input and gesture input.

[0056] Optionally, the first operation instruction may also be a first operation, including an instruction for determining a process for creating and adjusting a target orthogonal projection area.

[0057] The 2D display image is updated in response to receiving the second operation instruction.

[0058] The first operation instruction and the second operation instruction are obtained by recognizing the voice content and / or gesture issued by the target user.

[0059] Specifically, after projecting the complete 2D layer, the fusion display device can also receive a second operation instruction obtained by the sensor through the voice content and / or gesture recognition of the target user, and execute corresponding operations in turn according to the control instructions contained in the second operation instruction to complete the update of the visual content of the 2D display image.

[0060] The second operation instruction is used to adjust the rendering content of the task guidance element. The target user can select several virtual controls in the orthogonal projection plane to complete the size adjustment of the rendering content of the task guidance element. The adjustment scenarios include but are not limited to actively swiping away completed tasks, actively changing navigation methods, and actively browsing historical information. The embodiment of the present invention does not specifically limit this.

[0061] Optionally, the second operation instruction may be generated through touch input or key input of a physical control, as well as voice input and gesture input.

[0062] Optionally, the second operation instruction may also be a second operation, including an instruction for determining to create a process for adjusting the display content of a 2D display image.

[0063] In the embodiment of the present invention, the target orthogonal projection area and / or the 2D display image in the 2D display layer can be adjusted by receiving the perceived operation instruction, so as to update the 2D display content in real time. This can well ensure that the viewing area is expanded without affecting the display of the task guidance elements.

[0064] Based on any of the above embodiments, a 3D display image generated using real scene elements is placed in the first display layer of the user display interface in the augmented reality module, including: generating a 3D display image based on the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system.

[0065] Specifically, the fusion display device uses the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system to anchor the real space, and then combines it with the optical perspective image to generate a 3D display image for display enhancement.

[0066] The projection mode of the 3D display image is set to perspective projection, so that the 3D display image is displayed on the first display layer.

[0067] Specifically, the fusion display device sets the projection mode of the 3D display image to perspective projection, considers the X, Y and Z coordinates of all points in the three-dimensional space, and uses a perspective method to map the 3D display image corresponding to the real-time posture of the target user's head to the first display layer for rendering, so as to display the 3D display image in full screen.

[0068] The embodiment of the present invention combines the orientation information of the camera in the real scene and the position information of the elements of the real scene in the earth coordinate system for perspective projection, thereby obtaining a 3D display image for augmented reality or mixed reality of the elements of the real scene. The layer for 3D display can be placed below the 2D display layer, so that under the condition of limited hardware performance, some elements are displayed in 3D and the rest are displayed in 2D, so as to minimize the dizziness and discomfort caused by display time lag and improve the usability of equipment.

[0069] Based on any of the above embodiments, a 3D display image is generated according to the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system, including: drawing virtual objects on the first display layer according to the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system.

[0070] Specifically, the fusion display device draws virtual land features belonging to different real scene elements on the first display layer based on the relative position relationship between the real scene elements and the real environment calculated according to the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system.

[0071] The virtual objects are synthesized with the real scene images to obtain a 3D display image.

[0072] Specifically, the fusion display device synthesizes the virtual objects with the real scene images to obtain a 3D display image for augmented reality or mixed reality.

[0073] The embodiment of the present invention sets the arrangement of 2D display images and 3D display images according to the orientation information of the camera in the real scene and the position information of the elements of the real scene in the earth coordinate system, so that when the spatial position of the target user's eyes changes, the 3D display image is always in a picture with the best stereoscopic effect, and the imaging of the 3D display image in the target user's brain can always be integrated with the imaging of the real scene in the target user's brain.

[0074] Based on any of the above embodiments, the task guidance element includes one or more of a text box, a target plane map, navigation guidance information, and a video stream.

[0075] The target plane map includes a plane map centered on the target user and / or a three-dimensional map centered on the target object sighted by the augmented reality module.

[0076] Specifically, the fusion display device can visualize auxiliary intelligence information that is beneficial to combat and obtain mission guidance elements.

[0077] Optionally, the task guiding element may be a text box for displaying text information, which is not specifically limited in the embodiment of the present invention. For example, the text box may display text information exchanged between members who have established a communication connection, or the text box may display the task content.

[0078] Optionally, the task guidance element may be a target plane map, which is used to display scene information of the focus area, and the embodiment of the present invention does not specifically limit this. For example, the target plane map may be a 2D plane map of an area with the target user as the center and the alert distance as the radius, so that the target user can monitor the situation near him. Alternatively, the target plane map may also be a 3D stereoscopic map centered on the target object aimed at by the target user through the augmented reality module, so that the target user can monitor the situation near the target object when aiming at it.

[0079] Optionally, the task guidance element may be navigation guidance information for displaying position information, which is not specifically limited in the embodiments of the present invention. For example, the navigation guidance information may be a compass, so as to visualize the position information of the compass sensor itself obtained by sensing the strength of the earth's magnetic field. Alternatively, the navigation guidance information may also be a navigation ring, so as to mark the points marked by each team member who establishes a communication connection at the corresponding position in the navigation ring, so as to integrate all the position information that needs attention into one for visualization.

[0080] Optionally, the task guidance element may be a video stream for displaying real-time scene situation information, which is not specifically limited in the embodiment of the present invention. For example, the video stream may be video content publicly displayed by a team member who has established a communication connection. Alternatively, the video stream may also be video content collected in real time by the target user after aiming at the target object and observing the situation near the target object.

[0081] The embodiment of the present invention fixes elements such as text boxes, target floor maps, navigation guidance information, video streams, etc. that require the target user to pay close attention to and read in front of the target user in the form of a heads-up display, allowing information to be safely displayed to the target user without the target user having to look away from the content being observed in front.

[0082] Based on any of the above embodiments, the real scene elements include one or more of the marking points, member positions, assembly points, and the positions of the target objects aimed at by the augmented reality module.

[0083] Among them, the marking point has different marking graphics according to different marking sources.

[0084] Specifically, the fusion display device can visualize the real terrain information in the combat scene to obtain realistic scene elements.

[0085] Optionally, the real scene element may be a mark point, which is used to display the point that the target user needs to pay attention to, and the embodiment of the present invention does not specifically limit this. For example, the mark point can display the point shared by the members who have established a communication connection according to the battlefield situation, and different colors or graphics can be given to distinguish according to different marking sources, or the mark point can display the point indicated by the independent task of the target user himself.

[0086] Optionally, the real scene element may be a member position, which is used to display the real-time position of each member who has established a communication connection, so that the target user can learn about the distribution of teammates. This embodiment of the present invention does not make any specific limitation to this.

[0087] Optionally, the real scene element may be a rally point, which is used to establish a task point for consensus among members with communication connections to guide teammates to work together, and the embodiment of the present invention does not specifically limit this.

[0088] Optionally, the real scene elements may be one or more of the positions of the target objects aimed at by the augmented reality module, which are used as points of interest to the target user himself to guide the target user to complete an independent combat mission by himself. The embodiments of the present invention do not specifically limit this.

[0089] The embodiment of the present invention uses 3D augmented reality to display elements such as marker points, member positions, assembly points, and target objects aimed by the augmented reality module according to the real-time posture changes of the target user. The display content of necessary display elements can be adjusted according to the tracked head movement, and unnecessary elements are relatively fixed to both eyes for display. Although only part of the content is augmented reality, it also reduces equipment cost and power consumption.

[0090] Figure 2 is a schematic diagram of the structure of the fusion display device provided by the present invention. Based on any of the above embodiments, Figure 2 As shown, the device includes: an image acquisition module 210, a scene analysis module 220, an enhanced display module 230 and a fixed display module 240. Among them: The image acquisition module 210 is used to acquire the real scene image captured by the camera in the augmented reality module, and the augmented reality module includes a camera and a user display interface.

[0091] The scene analysis module 220 is used to determine the real scene elements based on the location information of the objects in the real scene image that changes with the location of the target user, and to determine the task guidance elements based on the intelligence information contained in the real scene image. The target user is the user of the augmented reality module.

[0092] The augmented display module 230 is used to place the 3D display image generated by using the real scene elements on the first display layer of the user display interface in the augmented reality module.

[0093] The fixed display module 240 is used to place the 2D display image generated by using the task guidance element on the second display layer of the user display interface in the augmented reality module.

[0094] The second display layer is superimposed on the first display layer.

[0095] Specifically, the image acquisition module 210, the scene analysis module 220, the enhanced display module 230 and the fixed display module 240 are electrically connected.

[0096] The scene analysis module 220 needs to classify the information elements to be displayed in the real scene image captured by the camera in the augmented reality module according to the actual task requirements of the target user. One type is the real scene element. When the position of the target user changes in the real world, this type of element will change with the scene changes perceived by the target user through the field of view of the augmented reality module. The other type is the task guidance element, which is the content that requires the target user's continuous attention and will not change with the scene changes in the field of view.

[0097] After the augmented display module 230 is aligned with the landmark feature through the augmented reality module, it uses the landmark feature as a real scene element, obtains the position information of the landmark feature in its field of view in the earth coordinate system, and then combines the position and posture information of the target user relative to each real scene element to solve the projection transformation model of the virtual object and the real scene. Finally, the virtual information corresponding to the real scene element is superimposed on the real scene image taken by the camera in the augmented reality module. The generated 3D display image is displayed in the first display layer at the bottom layer of the user display interface using an augmented reality or mixed reality display solution.

[0098] The fixed display module 240 can visualize the task guidance elements determined from the real scene image or the intelligence information contained in other devices to obtain a 2D display image, and place the obtained 2D display image in a second display layer superimposed on the first display layer in the user display interface for fixed display relative to the display.

[0099] Optionally, the fixed display module 240 includes a first visualization unit and a first projection unit, wherein: The first visualization unit is used to perform visualization processing on the task guidance elements belonging to the target category to generate a 2D display image.

[0100] The first projection unit is used to set the projection mode of the 2D display image to orthogonal projection, and display the 2D display image in a target orthogonal projection area in the second display layer.

[0101] The target category is any one of the categories possessed by the task guidance element; the target orthogonal projection area is an area at the edge of the second display layer, and the number of the target orthogonal projection areas is the same as the number of the target categories.

[0102] Optionally, the fixed display module 240 further includes an adjustment unit, wherein: the adjustment unit is configured to receive and respond to a first operation instruction to fold or unfold the target orthogonal projection area.

[0103] The 2D display image is updated in response to receiving the second operation instruction.

[0104] The first operation instruction and the second operation instruction are obtained by recognizing the voice content uttered by the target user and / or by recognizing the gestures of the target user.

[0105] Optionally, the enhanced display module 230 includes a second visualization unit and a second projection unit, wherein: The second visualization unit is used to generate a 3D display image according to the orientation information of the camera in the real scene and the position information of the elements of the real scene in the earth coordinate system.

[0106] The second projection unit is used to set the projection mode of the 3D display image to perspective projection, and display the 3D display image on the first display layer.

[0107] Optionally, the second projection unit includes a virtual rendering subunit and a reality fusion subunit, wherein: The virtual drawing subunit is used to draw virtual objects on the first display layer according to the orientation information of the camera in the real scene and the position information of the elements of the real scene in the earth coordinate system.

[0108] The reality fusion subunit is used to synthesize virtual objects with real scene images to obtain 3D display images.

[0109] Optionally, the task guidance element includes one or more of a text box, a target plane map, navigation guidance information, and a video stream.

[0110] The target plane map includes a plane map centered on the target user and / or a three-dimensional map centered on the target object sighted by the augmented reality module.

[0111] Optionally, the real scene elements include one or more of a marking point, a member position, a gathering point, and a position of a target object aimed at by the augmented reality module.

[0112] Among them, the marking point has different marking graphics according to different marking sources.

[0113] The fusion display device provided in the embodiment of the present invention is used to execute the above-mentioned fusion display method of the present invention. Its implementation method is consistent with the implementation method of the fusion display method provided by the present invention and can achieve the same beneficial effects, which will not be repeated here.

[0114] The embodiment of the present invention classifies the content that needs to be displayed in the real scene image captured by the augmented reality module into real scene elements and task guidance elements. The real scene elements that will change with the position of the target user are displayed in 3D, and the task guidance elements that will not change with the position of the target user are displayed in 2D and fixedly covered on the target area of ​​the 3D display image. The elements that require the target user to pay close attention to and read are designed as 2D and fixed in front of the eyes of the target user in the form of a head-up display. Since the movement of the head does not affect the position of these elements, the refresh rate of the near-eye display can be higher than 60Hz, which can effectively reduce the time lag of the motion trajectory of the coordinate system in the virtual world in the augmented reality module, and can effectively reduce the dizziness and discomfort caused by the time lag of the motion trajectory of the coordinate system in the virtual world in the augmented reality module, thereby reducing the equipment cost investment and power consumption on the basis of ensuring the availability of the augmented reality module.

[0115] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the fusion display method, which includes: obtaining a real scene image captured by a camera in an augmented reality module, wherein the augmented reality module includes a camera and a user display interface; determining a real scene element according to the location information of the object in the real scene image that changes with the location of the target user, determining a task guidance element according to the intelligence information contained in the real scene image, wherein the target user is a user of the augmented reality module; placing a 3D display image generated by using the real scene element in the first display layer of the user display interface in the augmented reality module; placing a 2D display image generated by using the task guidance element in the second display layer of the user display interface in the augmented reality module; wherein the second display layer is superimposed on the first display layer.

[0116] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fusion display method provided by the above-mentioned methods, and the method includes: obtaining a real scene image taken by a camera in an augmented reality module, and the augmented reality module includes a camera and a user display interface; determining real scene elements according to the location information of the objects in the real scene image that changes with the location of the target user, and determining task guidance elements according to the intelligence information contained in the real scene image, and the target user is the user of the augmented reality module; placing a 3D display image generated using the real scene elements in the first display layer of the user display interface in the augmented reality module; placing a 2D display image generated using the task guidance elements in the second display layer of the user display interface in the augmented reality module; wherein the second display layer is superimposed on the first display layer.

[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fusion display method provided by the above-mentioned methods, the method comprising: obtaining a real scene image taken by a camera in an augmented reality module, the augmented reality module comprising a camera and a user display interface; determining real scene elements based on the location information of objects in the real scene image that changes with the location of a target user, and determining task guidance elements based on the intelligence information contained in the real scene image, the target user being a user of the augmented reality module; placing a 3D display image generated using the real scene elements in a first display layer of the user display interface in the augmented reality module; placing a 2D display image generated using the task guidance elements in a second display layer of the user display interface in the augmented reality module; wherein the second display layer is superimposed on the first display layer.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fusion display method, characterized in that: include: Acquire a real scene image captured by a camera in an augmented reality module, wherein the augmented reality module includes a camera and a user display interface; Determine real scene elements according to the location information of the objects in the real scene image that changes with the location of the target user, and determine task guidance elements according to the intelligence information contained in the real scene image, wherein the target user is a user of the augmented reality module; Placing the 3D display image generated by using the real scene elements in the first display layer of the user display interface in the augmented reality module; Placing the 2D display image generated by using the task guiding element in the second display layer of the user display interface in the augmented reality module; The second display layer is superimposed on the first display layer.

2. The fusion display method according to claim 1, characterized in that: Placing the 2D display image generated by the task guiding element in the second display layer of the user display interface in the augmented reality module includes: Performing visualization processing on the task guidance elements belonging to the target category to generate the 2D display image; Setting the projection mode of the 2D display image to orthogonal projection, and displaying the 2D display image in a target orthogonal projection area in the second display layer; Among them, the target category is any one of the categories possessed by the task guidance element; the target orthogonal projection area is the area at the edge of the second display layer, and the number of the target orthogonal projection areas is the same as the number of the target categories.

3. The fusion display method according to claim 2, characterized in that: After setting the projection mode of the 2D display image to orthogonal projection and displaying the 2D display image in a target orthogonal projection area in the second display layer, the method further includes: receiving and responding to a first operation instruction, folding or unfolding the target orthogonal projection area; receiving and updating the 2D display image in response to a second operation instruction; The first operation instruction and the second operation instruction are obtained by recognizing the voice content uttered by the target user and / or by recognizing the gesture of the target user.

4. The fusion display method according to claim 1, characterized in that: Placing the 3D display image generated by using the real scene elements in the first display layer of the user display interface in the augmented reality module includes: Generate the 3D display image according to the orientation information of the camera in the real scene and the position information of the elements of the real scene in the earth coordinate system; The projection mode of the 3D display image is set to perspective projection, and the 3D display image is displayed on the first display layer.

5. The fusion display method according to claim 4, characterized in that: The generating the 3D display image according to the orientation information of the camera in the real scene and the position information of the real scene elements in the earth coordinate system includes: Drawing virtual objects on the first display layer according to the orientation information of the camera in the real scene and the position information of the elements of the real scene in the earth coordinate system; The virtual ground object is synthesized with the real scene image to obtain the 3D display image.

6. The fusion display method according to claim 1, characterized in that: The task guidance element includes one or more of a text box, a target plane map, navigation guidance information, and a video stream; The target plane map includes a plane map centered on the target user and / or a three-dimensional map centered on the target object sighted by the augmented reality module.

7. The fusion display method according to any one of claims 1 to 6, characterized in that: The real scene elements include one or more of a marking point, a member position, a rally point, and a position of a target object sighted by the augmented reality module; The marking points have different marking graphics according to different marking sources.

8. A fusion display device, characterized in that: include: An image acquisition module, used to acquire a real scene image captured by a camera in an augmented reality module, wherein the augmented reality module includes a camera and a user display interface; A scene analysis module, used to determine real scene elements according to the location information of the objects in the real scene image that changes with the location of the target user, and to determine the task guidance elements according to the intelligence information contained in the real scene image, wherein the target user is the user of the augmented reality module; An augmented display module, used to place the 3D display image generated by using the real scene elements in the first display layer of the user display interface in the augmented reality module; A fixed display module, used for placing the 2D display image generated by using the task guiding element in the second display layer of the user display interface in the augmented reality module; The second display layer is superimposed on the first display layer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the fusion display method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fusion display method according to any one of claims 1 to 7 is implemented.

Images