Image processing method, device, electronic device and storage medium

Abstract

The present application provides an image processing method and device, an electronic device and a storage medium, wherein firstly, a real target object is identified from an acquired real scene image, and a first virtual three-dimensional model corresponding to the real target object is obtained; then, the virtual target object to be displayed is determined according to the real scene image, and the relative posture relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the AR scene to be displayed is determined; finally, the first virtual three-dimensional model and the virtual target object are displayed in an AR scene containing the real target object according to the relative posture relationship. The present application displays virtual target objects and real target objects based on the relative posture relationship, and can achieve the occlusion of virtual target objects located behind real target objects, overcoming the defect in the prior art that real objects cannot occlude virtual objects located behind real objects.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to an image processing method and device, an electronic device and a storage medium. Background Art

[0002] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It simulates and processes information that is difficult to experience in the spatial range of the real world based on computers and other scientific technologies, forms virtual information, superimposes it on the real world, and is perceived by human senses, thereby achieving a sensory experience beyond reality.

[0003] In an AR scene, the real environment and virtual objects are superimposed and exist in the same screen and space. In the current AR scene display, there is a situation where virtual objects are displayed in front of real objects in the real environment, resulting in the real object being unable to block the virtual object that is already behind the real object. Summary of the invention

[0004] The embodiments of the present application at least provide an image processing method and device to display the occlusion effect of mutual occlusion between real objects and virtual objects in an AR scene.

[0005] The present application provides an image processing method, including:

[0006] Acquire real scene images;

[0007] Identify a real target object in the real scene image, and obtain a first virtual three-dimensional model corresponding to the real target object;

[0008] Determine a virtual target object, and determine a relative position relationship between a first virtual three-dimensional model corresponding to the real target object and the virtual target object in an augmented reality (AR) scene;

[0009] The first virtual three-dimensional model and the virtual target object are displayed in an AR scene including the real target object according to the relative posture relationship.

[0010] In this aspect, by utilizing the relative posture relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the AR scene, the real target object and the virtual target object are displayed in the AR scene, and the virtual target object located behind the real target object can be occluded, thereby overcoming the defect in the prior art that the real object cannot occlude the virtual object located behind the real object.

[0011] In a possible implementation manner, the display state of the first virtual three-dimensional model is a transparent state.

[0012] In this implementation, the virtual three-dimensional model corresponding to the real target object is set to be transparent, so that the first virtual three-dimensional model can better block the virtual target object located behind the real target object, while not affecting the display of objects on the layer in front of the layer where the first virtual three-dimensional model is located. The first virtual three-dimensional model is not displayed in the AR scene, and has no impact on the image display in the AR scene.

[0013] In a possible implementation, the acquiring a first virtual three-dimensional model corresponding to the real target object includes:

[0014] Acquire a prefabricated map; the prefabricated map includes images of multiple objects within a preset geographical range and object identification information of each object;

[0015] Matching the image of the real target object in the real scene image with the image of each of the multiple objects in the prefabricated map to obtain a target matching object that matches the real target object;

[0016] The object identification information of the target matching object is used as the object identification information of the real target object, and based on the object identification information, a first virtual three-dimensional model of the real target object which is preset is acquired.

[0017] This implementation method can achieve high-precision positioning of the real target object using the prefabricated map, so that the object identification information of the real target object can be accurately found, thereby improving the accuracy of obtaining the first virtual three-dimensional model.

[0018] In a possible implementation, determining the virtual target object includes:

[0019] Based on the real scene image, determining the position information of the shooting device that shoots the real scene image,

[0020] Based on the position information of the shooting device, a virtual target object in the shooting field of view of the shooting device is determined.

[0021] This implementation can achieve high-precision positioning of the shooting device through the real scene image, and based on the high-precision positioning information of the shooting device, that is, the above-mentioned posture information, it can more accurately determine the virtual target object to be displayed within the shooting field of view of the shooting device.

[0022] In a possible implementation manner, the above-mentioned image processing method further includes the step of setting the first virtual three-dimensional model:

[0023] Acquiring position information of the real target object in the real scene;

[0024] Determining the position and posture information of the real target object in the AR scene based on the position and posture information of the real target object in the real scene;

[0025] Based on the position information of the real target object in the AR scene, a first virtual three-dimensional model of the real target object in the AR scene is generated.

[0026] This implementation method converts the posture information between the real scene and the AR scene, and generates a virtual three-dimensional model that the real target object relies on for display in the AR scene, so that the real target object can be accurately displayed in the AR scene.

[0027] In a possible implementation, determining a relative position relationship between a first virtual three-dimensional model corresponding to the real target object and the virtual target object in an augmented reality (AR) scene includes:

[0028] Acquire a second virtual three-dimensional model corresponding to the virtual target object;

[0029] Based on the pose information of the first virtual three-dimensional model in the AR scene and the pose information of the second virtual three-dimensional model in the AR scene, the relative pose relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality AR scene is determined.

[0030] In this implementation, based on the pose information of the first virtual three-dimensional model in the AR scene and the pose information of the second virtual three-dimensional model in the AR scene, the relative pose relationship between the two can be determined more accurately.

[0031] In a possible implementation, displaying the first virtual three-dimensional model and the virtual target object according to the relative posture relationship in an AR scene including the real target object includes:

[0032] Based on the relative position and posture relationship, determining whether the virtual target object is blocked by the real target object;

[0033] In the case where the virtual target object is blocked by the real target object, determining, based on the relative position and posture relationship, blocking information corresponding to the blocking of the virtual target object by the first virtual three-dimensional model;

[0034] Based on the occlusion information, an unoccluded portion of the virtual target object is displayed.

[0035] This implementation, based on the relative posture relationship, can accurately determine whether occlusion occurs between the two, and can display the unobstructed virtual target object when occlusion occurs, thereby improving the accuracy of the occlusion effect display.

[0036] In a possible implementation, the image processing method further includes:

[0037] Acquire motion state information of the virtual target object in the AR scene;

[0038] Determine collision state information based on the position and posture information of the first virtual three-dimensional model in the AR scene, the position and posture information of the virtual target object in the AR scene, and the motion state information;

[0039] If the collision state information indicates that the real target object collides with the virtual target object, then based on the collision state information, obtaining a prefabricated third virtual three-dimensional model of the virtual target object in a collision state;

[0040] A virtual target object after the collision is presented in the AR scene based on the third virtual three-dimensional model.

[0041] This implementation method utilizes the motion state information of the virtual target object to determine the collision state of the real target object and the virtual target object, thereby obtaining a third virtual three-dimensional model after the collision between the two, so as to display the image of the virtual target object after the collision in the AR scene.

[0042] In a possible implementation, the virtual target object after the collision presented in the AR scene based on the third virtual three-dimensional model includes:

[0043] Acquire a prefabricated fourth virtual three-dimensional model of the real target object in a collision state;

[0044] Based on the fourth virtual three-dimensional model and the third virtual three-dimensional model, a real target object and a virtual target object after the collision are presented in the AR scene.

[0045] This implementation uses the fourth virtual three-dimensional model and the third virtual three-dimensional model in the collision state to more accurately display the image of the virtual target object and the image of the real target object after the collision in the AR scene, further improving the accuracy of the collision picture display.

[0046] In a possible implementation, the display states of the fourth virtual three-dimensional model and the third virtual three-dimensional model are both non-transparent, and the fourth virtual three-dimensional model and the third virtual three-dimensional model are located on the same layer, and the layer of the fourth virtual three-dimensional model is located in front of the layer where the image of the corresponding real target object in the AR scene is located.

[0047] This implementation further improves the authenticity and accuracy of the collision picture display by limiting the display states and layers of the fourth virtual three-dimensional model and the third virtual three-dimensional model.

[0048] In a possible implementation, the obtaining, based on the collision state information, a prefabricated third virtual three-dimensional model of the virtual target object in the collision state includes:

[0049] Based on the collision state information, determining a collision position and / or a collision force;

[0050] The third virtual three-dimensional model of the virtual target object that matches the collision position and / or the collision force is obtained.

[0051] This implementation utilizes the collision state information of the virtual target object to determine the collision position and collision force, so that the state of the two after the collision can be accurately displayed in the AR scene, further improving the authenticity and accuracy of the collision picture display.

[0052] The present application provides an image processing device, comprising:

[0053] An image acquisition module, used to acquire images of real scenes;

[0054] An object recognition module, used to recognize a real target object in the real scene image and obtain a first virtual three-dimensional model corresponding to the real target object;

[0055] A posture processing module, used to determine a virtual target object, and determine a relative posture relationship between the first virtual three-dimensional model and the virtual target object in an augmented reality (AR) scene;

[0056] An image display module is used to display the first virtual three-dimensional model and the virtual target object in an AR scene including the real target object according to the relative posture relationship.

[0057] An embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method described in the second aspect are performed.

[0058] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method described in the second aspect are executed.

[0059] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solutions of the present application. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0061] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application is shown;

[0062] Figure 2 A flowchart showing a collision state in an image processing method provided in an embodiment of the present application is shown;

[0063] Figure 3 A schematic diagram showing the structure of an image processing device provided in an embodiment of the present application is shown;

[0064] Figure 4 A schematic diagram showing a virtual target object provided by an embodiment of the present application being blocked by a real target object;

[0065] Figure 5 A schematic diagram showing a collision between a virtual target object and a real target object provided by an embodiment of the present application is shown;

[0066] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0067] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0068] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0069] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0070] In AR image display, after the client visual positioning is successful, the real objects and virtual objects in the AR scene will form a collision or occlusion effect. For example, in an AR scene, the virtual object behind the real object should be occluded by the real object. Currently, it is not possible to occlude virtual objects in AR scenes, and virtual objects behind real objects can also be seen, causing distortion of the AR scene.

[0071] In response to the above-mentioned defects, the present application discloses an image processing method and device, an electronic device and a storage medium, wherein the present application first identifies a real target object from an acquired real scene image, and obtains a first virtual three-dimensional model corresponding to the real target object; then, based on the real scene image, determines the virtual target object to be displayed, and determines the relative posture relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the AR scene to be displayed; finally, displays the first virtual three-dimensional model and the virtual target object in the AR scene containing the real target object according to the relative posture relationship. The present application displays virtual target objects and real target objects based on the relative posture relationship, and can achieve the occlusion of virtual target objects located behind real target objects, overcoming the defect in the prior art that real target objects cannot occlude virtual target objects located behind real target objects.

[0072] The image processing method, device, electronic device and storage medium disclosed in the present application are described below through specific embodiments.

[0073] like Figure 1 As shown, the embodiment of the present application discloses an image processing method, which can be applied to an AR device to display an image in which a virtual target object and a real target object are mutually occluded. Specifically, the image processing method includes the following steps:

[0074] S110, acquiring a real scene image.

[0075] Here, the real scene image can be obtained by using the camera of the AR device itself, or the AR device can obtain the real scene image from other shooting devices.

[0076] The real scene image includes all real objects within the field of view of the device that captured the image. The real objects may be buildings and other objects.

[0077] S120: Identify a real target object in the real scene image, and obtain a first virtual three-dimensional model corresponding to the real target object.

[0078] Here, after obtaining a real scene image including one or more real objects, image recognition technology can be used to identify it to obtain the real objects in the real scene image and the object identification information of the real objects. Afterwards, based on the object identification information, a first virtual three-dimensional model of a pre-set real object is obtained. Before performing this step, a virtual three-dimensional model is set in advance for each real object in the real scene, so as to render and display the corresponding real object in the AR scene based on the virtual three-dimensional model. After setting a virtual three-dimensional model for each real object, a mapping relationship is established between the object identification information of the real object and its corresponding virtual three-dimensional model, and the virtual three-dimensional model corresponding to the object identification information can be determined based on the mapping relationship.

[0079] In a specific implementation, the object identification information can be identified by the following steps: the image of the real object in the real scene image is matched with the image of each object in the prefabricated map to obtain a target matching object that matches the real object, and the target matching object is the real object in the prefabricated map. The prefabricated map includes images of multiple objects within a preset geographical range and object identification information of each object. Then, the object identification information of the real object can be obtained by using the target matching object obtained by matching.

[0080] The use of prefabricated maps can achieve high-precision positioning of real target objects, so that the object identification information of the real target objects can be accurately found, and a highly accurate first virtual three-dimensional model can be obtained to improve the accuracy of the real target objects displayed in the AR scene, and improve the accuracy of mutual occlusion between the real target objects and virtual target objects displayed in the AR scene.

[0081] S130: Determine a virtual target object, and determine a relative position relationship between a first virtual three-dimensional model corresponding to the real target object and the virtual target object in an augmented reality (AR) scene.

[0082] Here, the virtual target objects are pre-set, and before executing this step, the position information and appearance information of each virtual target object in the AR scene have been set. When executing this step, you only need to select the virtual target object that is within the field of view of the AR device or needs to be displayed in the AR scene from the multiple virtual objects that have been set.

[0083] In a specific implementation, visual positioning can be performed based on the real scene image to determine the posture information of the device that photographed the real scene image. Then, based on the posture information of the device, the virtual object in the shooting field of view of the device is determined to obtain the virtual target object to be displayed.

[0084] The real scene image can be used to achieve high-precision positioning of the device that captures the real scene image. Based on the high-precision positioning information and the above-mentioned posture information, the virtual target object to be displayed within the shooting field of view of the shooting device can be determined more accurately.

[0085] The relative position relationship between the first virtual three-dimensional model and the virtual target object in the AR scene to be displayed corresponds to the display state of the virtual target object in the AR scene after the first virtual three-dimensional model and the target virtual model occlude each other. Based on the relative position relationship, the display state of the virtual target object after being occluded by the first virtual three-dimensional model, that is, the real target object, can be displayed in the AR scene.

[0086] In this step, the first virtual three-dimensional model corresponding to the real target object can be set to a transparent state, and then the virtual target object and the real target object are displayed in the AR scene according to the above-mentioned relative posture relationship, so that the first virtual three-dimensional model can better block the virtual target object behind the real target object, while not affecting the display of objects on the layer in front of the layer where the first virtual three-dimensional model is located. The first virtual three-dimensional model is not displayed in the AR scene, and has no effect on the image display in the AR scene. The real target object is rendered in the AR scene according to the first virtual three-dimensional model, or the real target object is displayed in the AR scene as a background. Figure 4 As shown, using the image processing method of the present application, the virtual target object dinosaur 41 is partially blocked by the real target object building 42. Here, the first virtual 3D model displayed in a transparent state will not affect the display state of the real target object displayed in the background mode, and it will only block the virtual target object located behind the real target object.

[0087] Step 120 identifies the real target object in the real scene image and obtains the first virtual three-dimensional model corresponding to the real target object. Step 130 performs visual positioning of the shooting device and determines the virtual target object to be displayed accordingly. In specific implementation, the above steps can also be implemented in other orders. For example, the visual positioning of the shooting device can be performed first to determine the virtual target object to be displayed; then, the real target object in the real scene image is identified, and the first virtual three-dimensional model corresponding to the real target object is obtained and loaded. After the real target object is identified, the real target object can be rendered and displayed in the AR scene.

[0088] S140: Display the first virtual three-dimensional model and the virtual target object in an AR scene including the real target object according to the relative posture relationship.

[0089] Here, the relative posture relationship can be used to display the posture information or state of the virtual target object after the real target object and the virtual target object occlude each other in the AR scene. Specifically, when the virtual target object is partially occluded by the real target object in the AR scene, the posture information or state of the unoccluded part of the virtual target object is displayed. In addition, when the virtual target object is not occluded by the real target object, the entire posture information or state of the virtual target object is displayed.

[0090] In addition, all or part of the real target object is also displayed in the AR scene: specifically, in the AR scene, when the real target object is not blocked by the virtual target object, the full posture information or status of the real target object is displayed; when the real target object is partially blocked by the virtual target object, the posture information or status of the unblocked part of the real target object is displayed.

[0091] It can be seen from the above statements that the first virtual three-dimensional model is preset and can be implemented using the following steps:

[0092] Step 1: Obtain the position information of the real target object in the real scene.

[0093] Here, the posture information can be obtained by following the steps below:

[0094] Perform image recognition on a real target object to obtain object identification information of the real target object; use the object identification information to obtain posture information corresponding to the object identification information.

[0095] The above method for obtaining position and posture information requires that the position and posture information of the real target object be stored in advance and a mapping relationship be established with the object identification information.

[0096] Of course, other methods can also be used to obtain the posture information of the real target object in the real scene. For example, after visual positioning, it can be determined based on the posture information of the device that captures the real scene image and the posture information of the real target object in the real scene image.

[0097] Step 2: Based on the pose information of the real target object in the real scene, determine the pose information of the real target object in the AR scene.

[0098] Here, the transformation relationship between the coordinate system corresponding to the real scene and the coordinate system corresponding to the AR scene can be used to convert the posture information of the real target object in the real scene into the posture information of the real target object in the AR scene.

[0099] Step three: Based on the position information of the real target object in the AR scene, generate a first virtual three-dimensional model of the real target object in the AR scene.

[0100] Here, based on the position information of each pixel point on the real target object in the AR scene, a first virtual three-dimensional model corresponding to the real target object can be generated.

[0101] In the above-mentioned embodiment, the posture information is converted between the real scene and the AR scene, thereby generating a virtual three-dimensional model on which the real target object depends for display in the AR scene, so that the real target object can be displayed in the AR scene.

[0102] In some embodiments, the above-mentioned determination of the relative position relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality AR scene can be specifically implemented by the following steps:

[0103] Step 1: Obtain a second virtual three-dimensional model corresponding to the virtual target object.

[0104] The virtual target object in the AR scene is pre-set, and the virtual three-dimensional model on which it depends for display in the AR scene is also pre-set. Therefore, after the virtual target object is determined, the pre-stored second virtual three-dimensional model can be obtained. The second virtual three-dimensional model is generated according to the position information and appearance information of the corresponding virtual target object in the AR scene, and it is not transparent.

[0105] Step 2: Based on the pose information of the first virtual three-dimensional model in the AR scene and the pose information of the second virtual three-dimensional model in the AR scene, determine the relative pose relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality AR scene.

[0106] Based on the position and posture information of the first virtual three-dimensional model in the AR scene and the position and posture information of the second virtual three-dimensional model in the AR scene, the relative position and posture relationship between the two can be accurately determined.

[0107] In some embodiments, the above-mentioned displaying the first virtual three-dimensional model and the virtual target object according to the relative posture relationship in the AR scene including the real target object may specifically include the following steps:

[0108] Firstly, based on the relative posture relationship, determine whether the virtual target object is blocked by the real target object; secondly, in the case that the virtual target object is blocked by the real target object, based on the relative posture relationship, determine the occlusion information corresponding to the occlusion of the virtual target object by the first virtual three-dimensional model; finally, based on the occlusion information, display the unobstructed part of the virtual target object.

[0109] As described above, if it is determined that the virtual target object is partially or completely located behind the real target object based on the relative position and posture relationship, it is determined that the virtual target object is blocked by the real target object.

[0110] Based on the relative posture relationship, the posture information of the portion of the virtual target object that is blocked by the real target object can be determined. In a specific implementation, the posture information of the blocked portion can be used as the blocking information.

[0111] After determining the posture information of the obscured part of the virtual target object, the posture information of the unobstructed part of the virtual target object can be determined in combination with the overall posture information of the virtual target object. Then, the unobstructed part of the virtual target object can be displayed based on the posture information of the unobstructed part.

[0112] In addition, based on the relative posture relationship, the posture information of the part of the virtual target object that is not blocked by the real target object can be directly determined, and the unblocked part of the virtual target object can be displayed based on the posture information of the unblocked part.

[0113] The above-mentioned embodiment can accurately determine whether occlusion occurs between the two based on the relative posture relationship, and can display the unobstructed virtual target object when occlusion occurs, thereby improving the accuracy of the occlusion effect display.

[0114] The virtual target object can also be moving in the AR scene. During the movement of the virtual target object, it may collide with the real target object. In this case, it is necessary to display the state of the virtual target object and the real target object after the collision in the AR scene. Figure 2 As shown, the following steps can be used to implement collision display in AR scenes:

[0115] S210: Obtain motion state information of the virtual target object in the AR scene.

[0116] Here, the running state information may include information of the movement direction and movement speed of the virtual target object, etc. Based on the movement state information, it can be determined whether the virtual target object collides and when the collision occurs.

[0117] S220: Determine collision state information based on the position and posture information of the first virtual three-dimensional model in the AR scene, the position and posture information of the virtual target object in the AR scene, and the motion state information.

[0118] Here, the position information of the virtual target object in the AR scene can be updated based on the motion state information. When the position information of the virtual target object in the AR scene overlaps with the position information of the real target object in the AR scene, it is determined that the virtual target object and the real target object have collided, and the collision position of the collision can be determined. According to the movement speed of the virtual target object, the collision strength of the virtual target object and the real target object can also be determined. The above collision position and collision strength can be used as collision state information.

[0119] S230: If the collision state information indicates that the real target object collides with the virtual target object, based on the collision state information, obtain a prefabricated third virtual three-dimensional model of the virtual target object in a collision state.

[0120] This step may specifically include acquiring the third virtual three-dimensional model corresponding to the virtual target object based on the collision position and / or the collision force.

[0121] The above third virtual three-dimensional models are all pre-set, and different collision positions and different collision forces correspond to different third virtual three-dimensional models. A third virtual three-dimensional model also corresponds to a collision position and a collision force. The third virtual three-dimensional model has a preset mapping relationship with the collision position and the collision force, and the third virtual three-dimensional model corresponding to the collision position and the collision force can be screened out by using the mapping relationship.

[0122] Based on the third virtual three-dimensional model, a virtual target object after the collision can be rendered in the AR scene.

[0123] It can be seen from the above description that the motion state information of the virtual target object is used to determine the collision position and collision force, so that the state of the two after the collision can be accurately displayed in the AR scene.

[0124] S240: Based on the third virtual three-dimensional model, present a virtual target object after the collision in the AR scene.

[0125] When a collision occurs, the real target object may also change. In order to further improve the authenticity and accuracy of the collision picture, the image of the real target object after the collision needs to be displayed in the AR scene. This can be achieved by the following steps:

[0126] A prefabricated fourth virtual three-dimensional model of the real target object in a collision state is obtained; and based on the fourth virtual three-dimensional model and the third virtual three-dimensional model, the real target object and the virtual target object after the collision are presented in the AR scene.

[0127] The fourth virtual three-dimensional models are all pre-set, and different collision positions and different collision forces correspond to different fourth virtual three-dimensional models. A fourth virtual three-dimensional model also corresponds to a collision position and a collision force. The fourth virtual three-dimensional model has a preset mapping relationship with the collision position and the collision force, and the fourth virtual three-dimensional model corresponding to the collision position and the collision force can be screened out by using the mapping relationship.

[0128] The display states of the fourth virtual 3D model and the third virtual 3D model are both non-transparent, and the fourth virtual 3D model and the third virtual 3D model are located in the same layer, and the layer of the fourth virtual 3D model is located in front of the layer where the image of the corresponding real target object in the AR scene is located. By restricting the display states and layers of the fourth virtual 3D model and the third virtual 3D model, the authenticity and accuracy of the collision screen display are further improved.

[0129] In the above embodiment, based on the fourth virtual three-dimensional model and the third virtual three-dimensional model, the virtual target object and the real target object after the collision can be rendered in the AR scene, which further improves the accuracy of the collision picture display.

[0130] Of course, the real target object may also move. In this case, the collision state information is determined based on the position information of the first virtual 3D model in the AR scene, the motion state information of the real target object, and the position information of the virtual target object in the AR scene. Then, the virtual 3D model corresponding to the real target object and the virtual 3D model corresponding to the virtual target object are obtained based on the collision state information, and the state of the two after the collision is displayed in the AR scene using the obtained virtual 3D model. Figure 5 As shown, the real target object car 51 collides with the virtual target object water 52 during driving, and the state after the collision is as follows Figure 5 As shown, the state of the virtual object and the real object after the collision is truly displayed in the AR scene, realizing the combination of virtual and reality.

[0131] In the above embodiment, the real target object is modeled in advance to obtain the corresponding first virtual three-dimensional model; then, an AR device (mobile phone, etc.) is used to perform visual positioning in an AR guide application based on a high-precision map. After the visual positioning is successful, the real target object is recognized to enter the field of view of the AR device. The AR device obtains and loads the first virtual three-dimensional model, and displays the virtual object in the AR scene, which can achieve the collision and occlusion effects of the real object on the virtual object without distortion of the picture.

[0132] Corresponding to the above-mentioned image processing method, the present application also discloses an image processing device, which is applied to AR equipment, etc. Each module in the device can implement each step of the image processing method in each of the above-mentioned embodiments, and can achieve the same beneficial effects. Therefore, the same parts will not be described in detail here. Specifically, Figure 3 As shown, the image processing device includes:

[0133] The image acquisition module 310 is used to acquire images of real scenes.

[0134] The object recognition module 320 is used to recognize the real target object in the real scene image and obtain a first virtual three-dimensional model corresponding to the real target object.

[0135] The posture processing module 330 is used to determine a virtual target object, and determine a relative posture relationship between a first virtual three-dimensional model corresponding to the real target object and the virtual target object in an augmented reality (AR) scene.

[0136] The image display module 340 is used to display the first virtual three-dimensional model and the virtual target object in an AR scene including the real target object according to the relative posture relationship.

[0137] In some embodiments, the display state of the first virtual three-dimensional model is a transparent state.

[0138] In some embodiments, when acquiring the first virtual three-dimensional model corresponding to the real target object, the object recognition module 320 is used to:

[0139] Acquire a prefabricated map; the prefabricated map includes images of multiple objects within a preset geographical range and object identification information of each object;

[0140] Matching the image of the real target object in the real scene image with the image of each of the multiple objects in the prefabricated map to obtain a target matching object that matches the real target object;

[0141] The object identification information of the target matching object is used as the object identification information of the real target object, and based on the object identification information, a first virtual three-dimensional model of the real target object which is preset is acquired.

[0142] In some embodiments, when determining the virtual target object, the posture processing module 330 is used to:

[0143] Based on the real scene image, determining the position information of the shooting device that shoots the real scene image,

[0144] Based on the position information of the shooting device, a virtual target object in the shooting field of view of the shooting device is determined.

[0145] In some embodiments, the system further includes a model setting module 350 for setting the first virtual three-dimensional model. The model setting module 350 is used to:

[0146] Acquiring position information of the real target object in the real scene;

[0147] Determining the position and posture information of the real target object in the AR scene based on the position and posture information of the real target object in the real scene;

[0148] Based on the position information of the real target object in the AR scene, a first virtual three-dimensional model of the real target object in the AR scene is generated.

[0149] In some embodiments, when determining the relative posture relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality (AR) scene, the posture processing module 330 is used to:

[0150] Acquire a second virtual three-dimensional model corresponding to the virtual target object;

[0151] Based on the pose information of the first virtual three-dimensional model in the AR scene and the pose information of the second virtual three-dimensional model in the AR scene, the relative pose relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality AR scene is determined.

[0152] In some embodiments, when the image display module 340 displays the first virtual three-dimensional model and the virtual target object according to the relative posture relationship in the AR scene including the real target object, it is used to:

[0153] Based on the relative position and posture relationship, determining whether the virtual target object is blocked by the real target object;

[0154] In the case where the virtual target object is blocked by the real target object, determining, based on the relative position and posture relationship, blocking information corresponding to the blocking of the virtual target object by the first virtual three-dimensional model;

[0155] Based on the occlusion information, an unoccluded portion of the virtual target object is displayed.

[0156] In some embodiments, the object identification module 320 is further configured to:

[0157] Acquire motion state information of the virtual target object in the AR scene;

[0158] Determine collision state information based on the position and posture information of the first virtual three-dimensional model in the AR scene, the position and posture information of the virtual target object in the AR scene, and the motion state information;

[0159] If the collision state information indicates that the real target object collides with the virtual target object, then based on the collision state information, obtaining a prefabricated third virtual three-dimensional model of the virtual target object in a collision state;

[0160] The image display module 340 is further configured to present a virtual target object after the collision in the AR scene based on the third virtual three-dimensional model.

[0161] In some embodiments, the object recognition module 320 is further used to: obtain a prefabricated fourth virtual three-dimensional model of the real target object in a collision state;

[0162] The image display module 340 is further configured to present the real target object and the virtual target object after the collision in the AR scene based on the fourth virtual three-dimensional model and the third virtual three-dimensional model.

[0163] In some embodiments, the display states of the fourth virtual three-dimensional model and the third virtual three-dimensional model are both non-transparent, and the fourth virtual three-dimensional model and the third virtual three-dimensional model are located on the same layer, and the layer of the fourth virtual three-dimensional model is located in front of the layer where the image of the corresponding real target object in the AR scene is located.

[0164] In some embodiments, when the object recognition module 320 obtains the prefabricated third virtual three-dimensional model of the virtual target object in the collision state based on the collision state information, it is used to:

[0165] Based on the collision state information, determining a collision position and / or a collision force;

[0166] The third virtual three-dimensional model of the virtual target object that matches the collision position and / or the collision force is obtained.

[0167] Corresponding to the above-mentioned image processing method, the embodiment of the present application further provides an electronic device 600, such as Figure 6 FIG. 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, including:

[0168] Processor 61, memory 62, and bus 63; memory 62 is used to store execution instructions, including internal memory 621 and external memory 622; the internal memory 621 here is also called internal memory, which is used to temporarily store the calculation data in the processor 61, as well as the data exchanged with the external memory 622 such as a hard disk. The processor 61 exchanges data with the external memory 622 through the internal memory 621. When the electronic device 600 is running, the processor 61 and the memory 62 communicate through the bus 63, so that the processor 61 executes the following instructions: obtain a real scene image; identify a real target object in the real scene image, and obtain a first virtual three-dimensional model corresponding to the real target object; determine the virtual target object, and determine the relative posture relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality AR scene; display the first virtual three-dimensional model and the virtual target object in the AR scene containing the real target object according to the relative posture relationship.

[0169] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image processing method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0170] The computer program product of the multi-user collection method provided in the embodiment of the present application includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0171] The present application also provides a computer program, which implements any one of the methods of the aforementioned embodiments when executed by a processor. The computer program product can be implemented in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0172] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0175] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire real scene images; Identify a real target object in the real scene image, and obtain a first virtual three-dimensional model corresponding to the real target object; Based on the real scene image, determining the position and posture information of the shooting device that shot the real scene image; Based on the posture information of the shooting device, determine a virtual target object in the shooting field of view of the shooting device from a plurality of preset virtual objects, and determine a relative posture relationship between the first virtual three-dimensional model and the virtual target object in the augmented reality (AR) scene; Displaying the first virtual three-dimensional model and the virtual target object in an AR scene including the real target object according to the relative posture relationship; the display state of the first virtual three-dimensional model is a transparent state; wherein the first virtual three-dimensional model displayed in a transparent state does not affect the display state of the real target object displayed in a background manner; The method further comprises the step of setting the first virtual three-dimensional model: Performing image recognition on a real target object to obtain object identification information of the real target object; using the object identification information, obtaining position information of the real target object in a real scene; Determining the position and posture information of the real target object in the AR scene based on the position and posture information of the real target object in the real scene; Based on the position information of the real target object in the AR scene, generating a first virtual three-dimensional model of the real target object in the AR scene; the method further includes: Acquire motion state information of the virtual target object in the AR scene; Determine collision state information based on the position and posture information of the first virtual three-dimensional model in the AR scene, the position and posture information of the virtual target object in the AR scene, and the motion state information; If the collision state information indicates that the real target object collides with the virtual target object, then based on the collision state information, obtaining a prefabricated third virtual three-dimensional model of the virtual target object in a collision state; Acquire a prefabricated fourth virtual three-dimensional model of the real target object in a collision state; Based on the fourth virtual three-dimensional model and the third virtual three-dimensional model, a real target object and a virtual target object after the collision are presented in the AR scene; the display states of the fourth virtual three-dimensional model and the third virtual three-dimensional model are both non-transparent, and the fourth virtual three-dimensional model and the third virtual three-dimensional model are located in the same layer, and the layer of the fourth virtual three-dimensional model is located in front of the layer where the image of the corresponding real target object in the AR scene is located.

2. The image processing method according to claim 1, characterized in that: The obtaining of a first virtual three-dimensional model corresponding to the real target object comprises: Acquire a prefabricated map; the prefabricated map includes images of multiple objects within a preset geographical range and object identification information of each object; Matching the image of the real target object in the real scene image with the image of each of the multiple objects in the prefabricated map to obtain a target matching object that matches the real target object; The object identification information of the target matching object is used as the object identification information of the real target object, and based on the object identification information, a first virtual three-dimensional model of the real target object which is preset is acquired.

3. The image processing method according to claim 1 or 2, characterized in that: The determining of the relative position relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality (AR) scene includes: Acquire a second virtual three-dimensional model corresponding to the virtual target object; Based on the pose information of the first virtual three-dimensional model in the AR scene and the pose information of the second virtual three-dimensional model in the AR scene, the relative pose relationship between the first virtual three-dimensional model corresponding to the real target object and the virtual target object in the augmented reality AR scene is determined.

4. The image processing method according to claim 3, characterized in that: The step of displaying the first virtual three-dimensional model and the virtual target object in an AR scene including the real target object according to the relative posture relationship includes: Based on the relative position and posture relationship, determining whether the virtual target object is blocked by the real target object; In the case where the virtual target object is blocked by the real target object, determining, based on the relative position and posture relationship, blocking information corresponding to the blocking of the virtual target object by the first virtual three-dimensional model; Based on the occlusion information, an unoccluded portion of the virtual target object is displayed.

5. The image processing method according to claim 1, characterized in that: The step of acquiring a prefabricated third virtual three-dimensional model of the virtual target object in a collision state based on the collision state information comprises: Based on the collision state information, determining a collision position and / or a collision force; The third virtual three-dimensional model of the virtual target object that matches the collision position and / or the collision force is obtained.

6. An image processing device, characterized in that: include: An image acquisition module, used to acquire images of real scenes; An object recognition module, used to recognize a real target object in the real scene image and obtain a first virtual three-dimensional model corresponding to the real target object; A posture processing module is used to determine the posture information of a shooting device that shoots the real scene image based on the real scene image; based on the posture information of the shooting device, determine the virtual target object in the shooting field of view of the shooting device, and determine the relative posture relationship between the first virtual three-dimensional model and the virtual target object in the augmented reality AR scene; An image display module, used for displaying the first virtual three-dimensional model and the virtual target object in an AR scene including the real target object according to the relative posture relationship; the display state of the first virtual three-dimensional model is a transparent state; wherein the first virtual three-dimensional model displayed in a transparent state does not affect the display state of the real target object displayed in a background manner; The device further comprises a model setting module for setting the first virtual three-dimensional model, wherein the model setting module is used to: Performing image recognition on a real target object to obtain object identification information of the real target object; using the object identification information, obtaining position information of the real target object in a real scene; Determining the position and posture information of the real target object in the AR scene based on the position and posture information of the real target object in the real scene; Based on the position information of the real target object in the AR scene, a first virtual three-dimensional model of the real target object in the AR scene is generated; the object recognition module is also used to obtain the motion state information of the virtual target object in the AR scene; Determine collision state information based on the position and posture information of the first virtual three-dimensional model in the AR scene, the position and posture information of the virtual target object in the AR scene, and the motion state information; If the collision state information indicates that the real target object collides with the virtual target object, then based on the collision state information, obtaining a prefabricated third virtual three-dimensional model of the virtual target object in a collision state; Acquire a prefabricated fourth virtual three-dimensional model of the real target object in a collision state; The image display module is also used to present the real target object and the virtual target object after the collision in the AR scene based on the fourth virtual three-dimensional model and the third virtual three-dimensional model; the display states of the fourth virtual three-dimensional model and the third virtual three-dimensional model are both non-transparent, and the fourth virtual three-dimensional model and the third virtual three-dimensional model are located in the same layer, and the layer of the fourth virtual three-dimensional model is located in front of the layer where the image of the corresponding real target object in the AR scene is located.

7. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method as described in any one of claims 1 to 6 are performed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are executed.

Images