Model display method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a model display method and device, electronic equipment and storage medium, the method comprising: acquiring a projection image collected by a projection camera, and a thumbnail corresponding to each three-dimensional model included in the projection image; determining area information of an image superimposition area provided in the projection image; determining superimposition information of the thumbnail corresponding to the three-dimensional model based on projection information of the three-dimensional model in the projection image and the area information; wherein the superimposition information comprises a superimposition position and / or display size on the image superimposition area; and superimposing the thumbnail corresponding to each three-dimensional model on the image superimposition area of the projection image according to the superimposition information of the thumbnail, to generate a target display image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of three-dimensional model, and in particular, to a model display method and device, an electronic device and a storage medium. BACKGROUND

[0002] In a virtual reality scene, a digital twin project or the like, a user can perform a walkthrough of each three-dimensional model after the three-dimensional model is established. In the walkthrough of each three-dimensional model, two virtual cameras can be set, the two virtual cameras have the same position on a horizontal plane and different positions in a direction perpendicular to the horizontal plane, the two virtual cameras move synchronously, and in the moving process, a perspective view of the three-dimensional model is captured by using a first virtual camera and an overhead view of the three-dimensional model is captured by using a second virtual camera, thereby completing the walkthrough of each three-dimensional model.

[0003] However, the method of capturing images of the three-dimensional model by using the two virtual cameras has the problem of multi-camera synchronization, and the operation process is complicated and the efficiency is low. SUMMARY

[0004] Therefore, the present disclosure at least provides a model display method and device, an electronic device and a storage medium.

[0005] In a first aspect, the present disclosure provides a model display method, comprising:

[0006] obtaining a projection image captured by a projection camera and a thumbnail corresponding to each three-dimensional model included in the projection image;

[0007] determining region information of an image superimposition region set in the projection image;

[0008] determining superimposition information of the thumbnail corresponding to the three-dimensional model based on projection information of the three-dimensional model in the projection image and the region information, wherein the superimposition information comprises a superimposition position and / or a display size on the image superimposition region;

[0009] superimposing the thumbnail corresponding to each three-dimensional model on the image superimposition region of the projection image according to the superimposition information of the thumbnail, to generate a target display image.

[0010] In the method, a projection camera is used to collect a projection image, and a thumbnail corresponding to each three-dimensional model in the projection image is obtained. The thumbnail can represent the shape of the three-dimensional model. After determining the superimposition information corresponding to the thumbnail, the thumbnail corresponding to each three-dimensional model can be superimposed on the image superimposition area of the projection image according to the superimposition information corresponding to the thumbnail, to generate a target display image. The target display image includes both the projection image of the three-dimensional model and the thumbnail representing the shape of the three-dimensional model, realizing roaming of each three-dimensional model by using one virtual camera, relieving the problem of multi-camera synchronization, and improving the model roaming efficiency. Meanwhile, the projection image and the thumbnail are displayed by using one frame of target display image, relieving the problem of resource waste caused by multiple frames of images in the three-dimensional model roaming process.

[0011] In a possible implementation, generating the thumbnail corresponding to each three-dimensional model includes:

[0012] determining a plan view of the three-dimensional model in a target collection direction;

[0013] generating the thumbnail corresponding to the three-dimensional model based on the plan view.

[0014] Here, the plan view of the three-dimensional model in the target collection direction can accurately represent the shape of the three-dimensional model, and then the thumbnail corresponding to the three-dimensional model is generated based on the plan view, so that the thumbnail can accurately represent the three-dimensional model. Meanwhile, the thumbnail corresponding to the three-dimensional model is determined based on the plan view, so that the determination process of the thumbnail does not need manual hand-drawing, and the generation process of the thumbnail is relatively simple and efficient.

[0015] In a possible implementation, when the superimposition information includes a superimposition position, determining the superimposition information of the thumbnail corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projection image and the area information includes:

[0016] determining size ratio information between the image superimposition area and the projection image based on the area size indicated by the area information and the image size corresponding to the projection image;

[0017] determining the superimposition position of the thumbnail corresponding to the three-dimensional model based on the size ratio information and the position information of the three-dimensional model on the projection image indicated by the projection information corresponding to the three-dimensional model.

[0018] Here, by determining the size ratio information between the image superimposition region and the projection image, and further based on the position information of the three-dimensional model on the projection image, the superimposition position of the thumbnail corresponding to the three-dimensional model is determined, so that the relative position relationship of the plurality of three-dimensional models on the projection image and the relative position relationship of the thumbnails corresponding to the plurality of three-dimensional models on the image superimposition region are matched, and the accuracy of the determined superimposition position is improved.

[0019] In a possible implementation, the position information of the three-dimensional model on the projection image is determined by:

[0020] Based on the projection region information of the three-dimensional model in the projection image, the center point of the projection region corresponding to the three-dimensional model is determined, and the position information of the center point is determined as the position information of the three-dimensional model on the projection image; or,

[0021] The display position information of the three-dimensional model in the set world coordinate system is obtained.

[0022] The display position information is converted into the view coordinate system corresponding to the projection image by using the determined coordinate system conversion matrix, and the position information of the three-dimensional model on the projection image is obtained.

[0023] Here, the position information of the three-dimensional model on the projection image can be flexibly determined by the above two methods, which provides data support for subsequent determination of the superimposition position of the thumbnail corresponding to the three-dimensional model on the image superimposition region.

[0024] In a possible implementation, the superimposition position of the thumbnail corresponding to the three-dimensional model is determined based on the size ratio information and the position information of the three-dimensional model on the projection image indicated by the projection information corresponding to the three-dimensional model, including:

[0025] Based on the horizontal coordinate information in the position information of the three-dimensional model on the projection image and the width ratio in the size ratio information, the horizontal coordinate position in the superimposition position is determined.

[0026] Based on the vertical coordinate information in the position information of the three-dimensional model on the projection image and the height ratio in the size ratio information, the vertical coordinate position in the superimposition position is determined.

[0027] Here, the horizontal distance information of the thumbnail corresponding to the three-dimensional model is determined more simply and efficiently by the horizontal coordinate information in the position information of the three-dimensional model on the projection image and the width proportion in the size proportion information; and the vertical distance information of the thumbnail corresponding to the three-dimensional model is determined more simply and efficiently by the vertical coordinate information in the position information of the three-dimensional model on the projection image and the height proportion in the size proportion information; thereby realizing more accurate determination of the superimposition position of the thumbnail corresponding to each three-dimensional model by using the position information of each three-dimensional model in the projection image.

[0028] In a possible implementation, when the display size is included in the superimposition information, the determination of the superimposition information of the thumbnail corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projection image and the region information includes:

[0029] determining size proportion information between the image superimposition region and the projection image based on the region size indicated by the region information and the image size corresponding to the projection image;

[0030] determining the display size of the thumbnail corresponding to the three-dimensional model based on the size proportion information and the projection region information of the three-dimensional model in the projection image indicated by the projection information.

[0031] In the examples of the present disclosure, the display size of the thumbnail corresponding to the three-dimensional model is determined based on the size proportion information between the image superimposition region and the projection image and the projection region information of the three-dimensional model in the projection image indicated by the projection information, so that the proportion information between the display size of the thumbnail corresponding to the three-dimensional model and the image size of the three-dimensional model matches the size proportion information between the image superimposition region and the projection image, thereby improving the display effect of the thumbnail.

[0032] In a possible implementation, the determination of the display size of the thumbnail corresponding to the three-dimensional model based on the size proportion information and the projection region information of the three-dimensional model in the projection image indicated by the projection information includes:

[0033] determining a display width in the display size based on the width proportion indicated by the size proportion information and the width information in the projection region information;

[0034] determining a display height in the display size based on the height proportion indicated by the size proportion information and the height information in the projection region information.

[0035] In the examples of the present disclosure, by determining the size ratio information between the image superimposition region and the projection image, and then based on the projection region information of the three-dimensional model in the projection image indicated by the projection information, the display size of the thumbnail corresponding to the three-dimensional model is determined more accurately, so that the ratio between the size of the thumbnail corresponding to the three-dimensional model and the size of the three-dimensional model, and the ratio between the size of the image superimposition region and the size of the projection image are matched, and then the thumbnail corresponding to the three-dimensional model can be superimposed in the image superimposition region more accurately according to the display size.

[0036] The effects of the following devices, electronic devices, and the like are described in the above method, and will not be repeated here.

[0037] In a second aspect, the present disclosure provides a model display device, comprising:

[0038] An acquisition module is configured to acquire a projection image captured by a projection camera and a thumbnail corresponding to each three-dimensional model included in the projection image;

[0039] A first determination module is configured to determine region information of an image superimposition region arranged in the projection image;

[0040] A second determination module is configured to determine superimposition information of the thumbnail corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projection image and the region information, wherein the superimposition information includes a superimposition position and / or a display size on the image superimposition region;

[0041] A first generation module is configured to superimpose the thumbnail corresponding to each three-dimensional model in the image superimposition region of the projection image according to the superimposition information of the thumbnail, and generate a target display image.

[0042] In a third aspect, the present disclosure provides an electronic device, comprising a processor, a memory, and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the model display method according to the first aspect or any one of the embodiments.

[0043] In a fourth aspect, the present disclosure provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the model display method according to the first aspect or any one of the embodiments.

[0044] In order to make the above objectives, characteristics and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required to be used in the embodiments will be briefly introduced.

[0046] Figure 1 A flowchart of a model display method provided by an embodiment of the present disclosure is shown;

[0047] Figure 2 A schematic diagram of a plurality of collection directions corresponding to a three-dimensional model in a model display method provided by an embodiment of the present disclosure is shown;

[0048] Figure 3 A schematic diagram of a target display image provided by an embodiment of the present disclosure is shown;

[0049] Figure 4 An architectural schematic diagram of a model display device provided by an embodiment of the present disclosure is shown;

[0050] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure.

[0052] In a virtual reality scene, a digital twin project, etc., after a three-dimensional model is established, each three-dimensional model can be roamed. When each three-dimensional model is roamed, two virtual cameras can be set, the two virtual cameras are at the same position on a horizontal plane and are at different positions in a direction perpendicular to the horizontal plane, the two virtual cameras move synchronously, and in the process of moving, a perspective view of the three-dimensional model is collected by using a first virtual camera and an overhead view of the three-dimensional model is collected by using a second virtual camera, thereby completing the roaming of each three-dimensional model.

[0053] However, the way of collecting images of a three-dimensional model by using two virtual cameras has the problem of multi-camera synchronization, and the operation process is tedious and the efficiency is low. At the same time, the perspective view of the three-dimensional model collected by using the first virtual camera and the perspective view of the three-dimensional model collected by using the second virtual camera generate a plurality of frames of images with consistent resolution, which causes waste of resources.

[0054] In order to alleviate the above problems, the embodiments of the present disclosure provide a model display method, device, electronic device and storage medium.

[0055] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0056] To facilitate understanding of the embodiments of this disclosure, a model display method disclosed in this disclosure will first be described in detail. The execution entity of the model display method provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device or a server; the server may be a local server, a cloud server, etc. The terminal device may be a mobile phone, tablet, augmented reality (AR) glasses, a personal digital assistant (PDA), etc. In some possible implementations, the model display method can be implemented by a processor calling computer-readable instructions stored in memory.

[0057] See Figure 1 The diagram shown is a flowchart illustrating the model display method provided in this embodiment of the present disclosure. The method includes: S101-S104, specifically:

[0058] S101, acquire the projected image captured by the projection camera, and the thumbnail corresponding to each three-dimensional model included in the projected image;

[0059] S102, determine the region information of the image overlay region set in the projected image;

[0060] S103, based on the projection information of the three-dimensional model in the projected image and the region information, determine the overlay information of the thumbnail corresponding to the three-dimensional model; wherein, the overlay information includes: the overlay position and / or display size on the overlay area of ​​the image;

[0061] S104, according to the overlay information of the thumbnails, the thumbnails corresponding to each three-dimensional model are overlaid on the image overlay area of ​​the projected image to generate a target display image.

[0062] In the method, a projection image is collected by using a projection camera, and a thumbnail corresponding to each three-dimensional model in the projection image is obtained. The thumbnail can represent the shape of the three-dimensional model. After determining the superimposition information corresponding to the thumbnail, the thumbnail corresponding to each three-dimensional model can be superimposed on the image superimposition area of the projection image according to the superimposition information corresponding to the thumbnail, to generate a target display image. The target display image includes both the projection image of the three-dimensional model and the thumbnail representing the shape of the three-dimensional model, realizing roaming of each three-dimensional model by using a virtual camera, relieving the problem of multi-camera synchronization, and improving the model roaming efficiency. Meanwhile, the projection image and the thumbnail are displayed by using one frame of target display image, relieving the problem of resource waste caused by multiple frames of images in the three-dimensional model roaming process.

[0063] The following specifically describes S101-S104.

[0064] For S101:

[0065] The projection camera can be a virtual projection camera in a model rendering engine, or a virtual projection camera can be generated by calling a generated camera generation module, and the virtual projection camera serves as the projection camera. The projection camera can be a perspective projection camera or an orthogonal projection camera. When the projection camera is a perspective projection camera, the projection image is a perspective projection image. When the projection camera is an orthogonal projection camera, the projection image is an orthogonal projection image.

[0066] The projection camera corresponds to pose data, and the pose data includes camera position and camera pose. The camera position can be coordinate information of the projection camera in a world coordinate system corresponding to a virtual scene, and the camera pose can be Euler angle information of the projection camera.

[0067] In implementation, the projection camera can be used to collect a projection image of at least one three-dimensional model included in a virtual scene. After obtaining the projection image, a thumbnail corresponding to each three-dimensional model in the projection image is obtained. The thumbnail corresponding to each three-dimensional model can be generated in advance before the projection image is collected, or can be generated in real time after the projection image is collected. The thumbnail corresponding to the three-dimensional model can be an image that can represent the shape of the three-dimensional model. For example, the thumbnail of the three-dimensional model can be an elevation view or a top view of the three-dimensional model.

[0068] In an implementable manner, generating a thumbnail corresponding to each three-dimensional model can include: determining a plan view of the three-dimensional model in a target collection direction; and generating the thumbnail corresponding to the three-dimensional model based on the plan view.

[0069] The target collection direction corresponding to the three-dimensional model can be set as needed. For example, one three-dimensional model can correspond to one target collection direction, or different three-dimensional models can correspond to different target collection directions. For example, when different models correspond to one target collection direction, the target collection direction can be any one of the top-down direction, the upward direction, the front view direction, the back view direction, the left side view direction, and the right side view direction. Referring to Figure 2 The schematic diagram of the plurality of collection directions is shown.

[0070] When different models correspond to different target collection directions, the target collection direction corresponding to the three-dimensional model can be determined according to the structural characteristics of the three-dimensional model. For example, if the front view direction of the three-dimensional model 1 has an identifying feature, the target collection direction corresponding to the three-dimensional model 1 can be the front view direction; if the top-down direction of the three-dimensional model 2 has an identifying feature, the target collection direction corresponding to the three-dimensional model 2 can be the top-down direction, and so on.

[0071] After the target collection direction of the three-dimensional model is determined, the plan view of the three-dimensional model in the target collection direction can be determined. For example, a newly generated virtual camera can be used to collect the plan view of the three-dimensional model in the target collection direction. Alternatively, a projection camera can be used to collect the plan view of the three-dimensional model in the target collection direction.

[0072] In implementation, the plan view can be used as the thumbnail corresponding to the three-dimensional model. Alternatively, the size, brightness, and other pixel information of the plan view can be adjusted, and the adjusted plan view can be used as the thumbnail corresponding to the three-dimensional model.

[0073] Here, by determining the plan view of the three-dimensional model in the target collection direction, the plan view of the three-dimensional model in the target collection direction can accurately represent the shape of the three-dimensional model, and then based on the plan view, the thumbnail corresponding to the three-dimensional model is generated, and the thumbnail can accurately represent the three-dimensional model. At the same time, by determining the thumbnail corresponding to the three-dimensional model based on the plan view, the determination process of the thumbnail does not require manual hand-drawing, and the generation process of the thumbnail is relatively simple and efficient.

[0074] For S102:

[0075] In implementation, an image superimposition area can be set in the projection image, so as to subsequently superimpose the adjusted thumbnail in the image superimposition area. The size of the image superimposition area can be determined according to the size of the projection image. For example, the size of the image superimposition area can be one-tenth, one-twentieth, or the like of the size of the projection image. That is, if the size of the projection image is 510x510, the size of the image superimposition area can be 51x51.

[0076] Alternatively, the proportional relationship between the width and the height of the projection image can also be determined according to the size information of the projection image; and the size of the image superimposition region is determined based on the proportional relationship. For example, when the size of the projection image is 1024x1024, the size of the image superimposition region can be 128x128, 64x64, etc., and when the size of the projection image is 1024x512, the size of the image superimposition region can be 128x64, 64x32, etc.

[0077] The position of the image superimposition region can be determined according to the position of the blank region included in the projection image. For example, when the blank region at the top-left position of the projection image is larger, a specific-size image superimposition region can be set at the top-left position of the projection image. Alternatively, any one of the top-left position, the bottom-left position, the top-right position, and the bottom-right position of the projection image can be selected as the position of the image superimposition region.

[0078] After the image superimposition region is set in the projection image, the region information of the image superimposition region in the projection image can be determined. The region information can include the position information and the size information of the image superimposition region in the projection image.

[0079] For S103:

[0080] After generating the thumbnail corresponding to each three-dimensional model, the thumbnail is superimposed on the image superimposition region of the projection image. When the thumbnail is superimposed on the image superimposition region, the display size and / or the superimposition position of the thumbnail on the image superimposition region need to be determined.

[0081] In implementation, the superimposition position and / or the display size of the thumbnail corresponding to the three-dimensional model on the image superimposition region are determined based on the projection information of the three-dimensional model in the projection image and the region information of the image superimposition region. The projection information can include the image size of the three-dimensional model in the projection image and the position information of the three-dimensional model on the projection image, i.e., the coordinate information in the view coordinate system corresponding to the projection image. The region information can include the region size of the image superimposition region.

[0082] For example, the image proportion occupied by the three-dimensional model in the projection image can be determined according to the image size of the three-dimensional model in the projection image and the size information of the projection image; and the display size of the thumbnail corresponding to the three-dimensional model is determined according to the image proportion occupied by the three-dimensional model in the projection image and the region size of the image superimposition region, so that the image proportion occupied by the three-dimensional model in the projection image matches the region proportion occupied by the thumbnail corresponding to the three-dimensional model in the image superimposition region.

[0083] For example, the superimposition position of the thumbnail corresponding to the three-dimensional model on the image projection region can be determined according to the position information of the three-dimensional model on the projection image and the size ratio between the projection image and the image superimposition region, so that the position of the three-dimensional model in the projection image matches the position of the thumbnail corresponding to the three-dimensional model in the image superimposition region.

[0084] In an implementation, when the superimposition information includes a superimposition position, the determining of the superimposition information of the thumbnail corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projection image and the region information can include:

[0085] In step A1, the size ratio information between the image superimposition region and the projection image is determined based on the region size indicated by the region information of the image superimposition region and the image size corresponding to the projection image. The size ratio information can include a width ratio and a height ratio. For example, a first ratio between the width value in the region size of the image superimposition region and the width value in the image size of the projection image is determined as the width ratio, and a second ratio between the height value in the region size of the image superimposition region and the height value in the image size of the projection image is determined as the height ratio.

[0086] In step A2, the position information of the three-dimensional model on the projection image is determined first, and then the superimposition position of the thumbnail corresponding to the three-dimensional model is determined based on the size ratio information and the position information of the three-dimensional model on the projection image.

[0087] In step A1, the size ratio information between the image superimposition region and the projection image is determined based on the region size indicated by the region information of the image superimposition region and the image size corresponding to the projection image. The size ratio information can include a width ratio and a height ratio. For example, a first ratio between the width value in the region size of the image superimposition region and the width value in the image size of the projection image is determined as the width ratio, and a second ratio between the height value in the region size of the image superimposition region and the height value in the image size of the projection image is determined as the height ratio.

[0088] In step A2, the position information of the three-dimensional model on the projection image is determined first, and then the superimposition position of the thumbnail corresponding to the three-dimensional model is determined based on the size ratio information and the position information of the three-dimensional model on the projection image.

[0089] For example, the superimposition position of the thumbnail corresponding to the three-dimensional model can be determined by multiplying the position information and the size ratio information, so that a first ratio between the distance from the position of the three-dimensional model to the lower boundary of the projection image and the height of the projection image matches a second ratio between the distance from the superimposition position of the thumbnail to the lower boundary of the image superimposition region and the height of the image superimposition region, and a third ratio between the distance from the position of the three-dimensional model to the left boundary of the projection image and the width of the projection image matches a fourth ratio between the distance from the superimposition position of the thumbnail to the left boundary of the image superimposition region and the width of the image superimposition region.

[0090] For example, referring to Figure 3 As shown, the width value in the area size of the image superimposition area is a1, the height value is b1, the width value of the image size corresponding to the projection image is a2, and the height value is b2. The width ratio in the size ratio information between the image superimposition area and the projection image is a1 / a2, and the height ratio is b1 / b2. The position information of the three-dimensional model D on the projection image is determined; based on the size ratio information and the position information of the three-dimensional model D on the projection image, the superimposition position of the thumbnail D' corresponding to the three-dimensional model is determined.

[0091] Here, by determining the size ratio information between the image superimposition area and the projection image, and then determining the superimposition position of the thumbnail corresponding to the three-dimensional model based on the position information of the three-dimensional model on the projection image, the relative positional relationship of the plurality of three-dimensional models on the projection image is matched with the relative positional relationship of the thumbnails corresponding to the plurality of three-dimensional models on the image superimposition area, and the accuracy of the determined superimposition position is improved.

[0092] In an implementation manner, the position information of the three-dimensional model on the projection image can be determined according to the following two manners:

[0093] Manner one, based on the projection area information of the three-dimensional model in the projection image, the center point of the projection area corresponding to the three-dimensional model is determined, and the position information of the center point is determined as the position information of the three-dimensional model on the projection image.

[0094] Manner two, the display position information of the three-dimensional model in the set world coordinate system is obtained; the display position information is converted to the view coordinate system corresponding to the projection image by using the determined coordinate system conversion matrix, to obtain the position information of the three-dimensional model on the projection image.

[0095] In the manner one, the projection area information of the three-dimensional model in the projection image is determined, which can be the contour area of the three-dimensional model on the projection image (the contour area can be the projection area), or a detection box corresponding to the contour area on the projection image (the area where the detection box is located can be the projection area); that is, the projection area information can include the position information of the contour area on the projection image, or the position information of the detection box corresponding to the contour area on the projection image, etc.

[0096] The center point of the projection area corresponding to the three-dimensional model can be further determined, and the position information of the center point is determined as the position information of the three-dimensional model on the projection image.

[0097] In the second mode, generally, after the three-dimensional model is constructed, a display pose of the three-dimensional model in the real scene can be determined, the display pose including display position information and pose information. Therefore, the display position information of the three-dimensional model in the set world coordinate system can be obtained. The world coordinate system can be a coordinate system constructed with a target scene position in the real scene as an origin.

[0098] In implementation, the display position information of the three-dimensional model in the world coordinate system can be converted to the camera coordinate system corresponding to the projection camera by using the determined first coordinate system conversion matrix to obtain intermediate position information. Then, the intermediate display position information of the three-dimensional model in the camera coordinate system corresponding to the projection camera can be converted to the view coordinate system corresponding to the projection image by using the second coordinate system conversion matrix to obtain the position information of the three-dimensional model on the projection image. The first coordinate system conversion matrix can be determined according to the camera extrinsic parameter, the position of the projection camera, and the like. The second coordinate system conversion matrix can be determined according to the camera intrinsic parameter, the position of the projection camera, and the like.

[0099] Here, the position information of the three-dimensional model on the projection image can be flexibly determined by the above two modes to provide data support for subsequent determination of the superimposition position of the thumbnail corresponding to the three-dimensional model on the image superimposition region.

[0100] In an implementation mode, in step A2, based on the size ratio information and the position information of the three-dimensional model on the projection image indicated by the projection information corresponding to the three-dimensional model, the superimposition position of the thumbnail corresponding to the three-dimensional model can be determined, which can include:

[0101] The horizontal coordinate position in the superimposition position is determined based on the horizontal coordinate information in the position information of the three-dimensional model on the projection image and the width ratio in the size ratio information.

[0102] The vertical coordinate position in the superimposition position is determined based on the vertical coordinate information in the position information of the three-dimensional model on the projection image and the height ratio in the size ratio information.

[0103] In implementation, after the size ratio information between the image superimposition region and the projection image and the position information of the three-dimensional model on the projection image are determined, the horizontal distance information of the thumbnail corresponding to the three-dimensional model from the left boundary of the image superimposition region can be determined based on the horizontal coordinate information in the position information of the three-dimensional model on the projection image and the width ratio in the size ratio information. Then, the horizontal coordinate position in the superimposition position can be determined based on the horizontal distance information and the region information.

[0104] Based on the ordinate information of the 3D model's position on the projected image and the height ratio in the size scale information, the distance between the thumbnail corresponding to the 3D model and the lower boundary of the image overlay area is determined, i.e., the vertical distance information. Furthermore, based on the vertical distance information and the area information, the ordinate position in the overlay location can be determined.

[0105] The horizontal and vertical coordinates can be coordinates in the view coordinate system corresponding to the projected image, or they can be coordinates in the view coordinate system corresponding to the image overlay area.

[0106] For example, see Figure 3 As shown, the position information of the 3D model D in the view coordinate system corresponding to the projected image is (768, 128), that is, the horizontal coordinate information is 768 and the vertical coordinate information is 128; the width ratio in the size scale information is a1 / a2, that is, the width ratio is 1 / 16; the height ratio in the size scale information is b1 / b2, that is, the height ratio is 1 / 16.

[0107] Multiplying the width ratio of 1 / 16 in the size scale information by the horizontal coordinate information (768) in the position information of the 3D model D, we get a horizontal distance of 48; multiplying the height ratio of 1 / 16 in the size scale information by the vertical coordinate information (128) in the position information of the 3D model D, we get a vertical distance of 8; then, based on the determined horizontal distance information, vertical distance information, and region information, we can determine the horizontal and vertical coordinate positions of the superposition location. For example, in Figure 3 In the image, the overlay position of the thumbnail corresponding to the 3D model D can be (48, 488), which is the position information in the view coordinate system corresponding to the projected image.

[0108] Here, by using the horizontal coordinate information of the 3D model's position information on the projected image and the width ratio in the size scale information, the horizontal distance information of the thumbnail corresponding to the 3D model can be determined relatively easily and efficiently; and by using the vertical coordinate information of the 3D model's position information on the projected image and the height ratio in the size scale information, the vertical distance information of the thumbnail corresponding to the 3D model can be determined relatively easily and efficiently; thus, by utilizing the position information of each 3D model in the projected image, the superposition position of the thumbnail corresponding to each 3D model can be determined more accurately.

[0109] In one possible implementation, when the overlay information includes the display size, determining the overlay information of the thumbnail corresponding to the 3D model based on the projection information of the 3D model in the projected image and the region information may include:

[0110] Step B1, determining size ratio information between the image superimposition region and the projection image based on the region size indicated by the region information and the image size corresponding to the projection image;

[0111] Step B2, determining the display size of the thumbnail corresponding to the three-dimensional model based on the size ratio information and the projection region information of the three-dimensional model in the projection image indicated by the projection information.

[0112] In step B1, the size ratio information between the image superimposition region and the projection image can be determined based on the region size indicated by the region information of the image superimposition region and the image size corresponding to the projection image; wherein the size ratio information includes width ratio and height ratio. Specifically, the determination process of the size ratio information can refer to the description of step A1, which will not be described in detail here.

[0113] In step B2, the display size of the thumbnail corresponding to each three-dimensional model is determined based on the size ratio information and the projection region information of each three-dimensional model in the projection image indicated by the projection information. Wherein the projection region information can include the detection box corresponding to the contour region of the three-dimensional model on the projection image, and the size information on the projection image.

[0114] For example, target detection can be performed on the projection image to determine the projection region information corresponding to each three-dimensional model in the projection image. For example, a trained neural network can be used to perform target detection on the projection image.

[0115] For example, the width value indicated by the projection region information can be multiplied by the width ratio in the size ratio information to obtain the width value in the display size; the height value indicated by the projection region information can be multiplied by the height ratio in the size ratio information to obtain the height value in the display size.

[0116] For example, referring to FIG. 1, Figure 3 As shown in FIG. 1, the width ratio in the size ratio information between the image superimposition region and the projection image is a1 / a2, and the height ratio is b1 / b2; the projection region information corresponding to the three-dimensional model D includes a width of m2 and a height of n2, and the display size of the thumbnail D' corresponding to the three-dimensional model can be determined. That is, the display size can be width: a1 / a2 x m2, and height: b1 / b2 x n2.

[0117] In the examples of the present disclosure, by determining the size ratio information between the image superimposition region and the projection image, and based on the projection region information of the three-dimensional model in the projection image indicated by the projection information, the display size of the thumbnail corresponding to the three-dimensional model is determined, so that the ratio information between the display size of the thumbnail corresponding to the three-dimensional model and the image size of the three-dimensional model matches the size ratio information between the image superimposition region and the projection image, and the display effect of the thumbnail is improved.

[0118] In an implementable manner, the display size of the thumbnail corresponding to the three-dimensional model can be determined based on the size ratio information and the projection region information of the three-dimensional model in the projection image indicated by the projection information, which can include:

[0119] determining the display width in the display size based on the width ratio indicated by the size ratio information and the width information in the projection region information;

[0120] determining the display height in the display size based on the height ratio indicated by the size ratio information and the height information in the projection region information.

[0121] determining the display width in the display size based on the width ratio indicated by the size ratio information and the width information in the projection region information; for example, the width information in the projection region information can be multiplied by the width ratio to obtain the display width; and determining the display height in the display size based on the height ratio indicated by the size ratio information and the height information in the projection region information; for example, the height information in the projection region information can be multiplied by the height ratio to obtain the display height.

[0122] For example, referring to Figure 3 As shown in the figure, the width of the three-dimensional model D in the projection image indicated by the projection information is m2, and the height is n2; the width ratio in the size ratio information is a1 / a2, and the height ratio in the size ratio information is b1 / b2.

[0123] The width ratio a1 / a2 in the size ratio information is multiplied by the width m2 of the three-dimensional model D, that is, (1 / 16)×64, to obtain the display width of the thumbnail corresponding to the three-dimensional model as 4; the height ratio b1 / b2 in the size ratio information is multiplied by the height n2 of the three-dimensional model D, that is, (1 / 16)×16, to obtain the display height of the thumbnail corresponding to the three-dimensional model as 1; that is, the display size of the thumbnail D' corresponding to the three-dimensional model D can be determined as: the display width is 4, and the display height is 1.

[0124] In the examples of the present disclosure, by determining the size ratio information between the image superimposition region and the projection image, and then based on the projection region information of the three-dimensional model in the projection image indicated by the projection information, the display size of the thumbnail corresponding to the three-dimensional model is determined more accurately, so that the ratio between the size of the thumbnail corresponding to the three-dimensional model and the size of the three-dimensional model, and the ratio between the size of the image superimposition region and the size of the projection image are matched, and then the thumbnail corresponding to the three-dimensional model can be superimposed in the image superimposition region more accurately according to the display size.

[0125] For S104:

[0126] After determining the superimposition position and / or display size of the thumbnail corresponding to each three-dimensional model in the image superimposition region, the thumbnail corresponding to each three-dimensional model can be superimposed on the image superimposition region of the projection image according to the superimposition position and / or display size of the thumbnail, to obtain a target display image.

[0127] For example, when the superimposition information includes the superimposition position and does not include the display size, the thumbnail corresponding to the three-dimensional model can be first adjusted to a preset size to obtain an adjusted thumbnail; and then the adjusted thumbnail is superimposed on the superimposition position of the image superimposition region to generate the target display image.

[0128] For another example, when the superimposition information does not include the superimposition position and includes the display size, the size of the thumbnail corresponding to the three-dimensional model can be first adjusted to the display size to obtain an adjusted thumbnail; and then the adjusted thumbnail is superimposed on a preset position of the image superimposition region to generate the target display image. The preset size and the preset position can be set as needed.

[0129] For another example, when the superimposition information includes the superimposition position and the display size, the thumbnail corresponding to the three-dimensional model can be first adjusted to the display size to obtain an adjusted thumbnail; and then the adjusted thumbnail is superimposed on the superimposition position of the image superimposition region to generate the target display image.

[0130] In implementation, after the target display image is generated, the display screen of any device can be controlled to display the target display image. The device can be a mobile phone, a computer, etc.

[0131] Those skilled in the art can understand that, in the above method of the specific implementation manner, the writing order of the steps does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of the steps should be determined by its function and possible internal logic.

[0132] Based on the same concept, the present disclosure also provides a model display device, which is described with reference to Figure 4As shown, the architecture schematic diagram of the model display device provided by the embodiment of the present disclosure includes an acquisition module 401, a first determination module 402, a second determination module 403, and a first generation module 404, and specifically:

[0133] The acquisition module 401 is configured to acquire a projection image collected by a projection camera and a thumbnail corresponding to each three-dimensional model included in the projection image.

[0134] The first determination module 402 is configured to determine region information of an image superimposition region arranged in the projection image.

[0135] The second determination module 403 is configured to determine superimposition information of the thumbnail corresponding to the three-dimensional model based on projection information of the three-dimensional model in the projection image and the region information, wherein the superimposition information includes a superimposition position and / or a display size on the image superimposition region.

[0136] The first generation module 404 is configured to superimpose the thumbnail corresponding to each three-dimensional model on the image superimposition region of the projection image according to the superimposition information of the thumbnail, and generate a target display image.

[0137] In a possible implementation, the device further includes a second generation module 405, which is configured to generate the thumbnail corresponding to each three-dimensional model according to the following steps:

[0138] Determine a plan view of the three-dimensional model in a target collection direction.

[0139] Generate the thumbnail corresponding to the three-dimensional model based on the plan view.

[0140] In a possible implementation, when the superimposition information includes the superimposition position, the second determination module 403, when determining the superimposition information of the thumbnail corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projection image and the region information, is configured to:

[0141] Determine size proportion information between the image superimposition region and the projection image based on a region size indicated by the region information and an image size corresponding to the projection image.

[0142] Determine the superimposition position of the thumbnail corresponding to the three-dimensional model based on the size proportion information and position information of the three-dimensional model on the projection image indicated by the projection information corresponding to the three-dimensional model.

[0143] In a possible implementation, when determining the position information of the three-dimensional model on the projection image, the second determination module 403 is configured to:

[0144] determine a center point of the projection region corresponding to the three-dimensional model based on the projection region information of the three-dimensional model in the projection image, and determine the position information of the center point as the position information of the three-dimensional model on the projection image; or

[0145] obtain display position information of the three-dimensional model in a set world coordinate system;

[0146] convert the display position information to a view coordinate system corresponding to the projection image by using the determined coordinate system conversion matrix, to obtain the position information of the three-dimensional model on the projection image.

[0147] In a possible implementation, the second determining module 403, when determining the superimposition position of the thumbnail corresponding to the three-dimensional model based on the size ratio information and the position information of the three-dimensional model on the projection image indicated by the projection information, is configured to:

[0148] determine the horizontal coordinate position in the superimposition position based on the horizontal coordinate information in the position information of the three-dimensional model on the projection image and the width ratio in the size ratio information;

[0149] determine the vertical coordinate position in the superimposition position based on the vertical coordinate information in the position information of the three-dimensional model on the projection image and the height ratio in the size ratio information.

[0150] In a possible implementation, the second determining module 403, when determining the superimposition information of the thumbnail corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projection image and the region information in the case where the superimposition information includes display size, is configured to:

[0151] determine the size ratio information between the image superimposition region and the projection image based on the region size indicated by the region information and the image size corresponding to the projection image;

[0152] determine the display size of the thumbnail corresponding to the three-dimensional model based on the size ratio information and the projection region information of the three-dimensional model in the projection image indicated by the projection information.

[0153] In a possible implementation, the second determining module 403, when determining the display size of the thumbnail corresponding to the three-dimensional model based on the size ratio information and the projection region information of the three-dimensional model in the projection image indicated by the projection information, is configured to:

[0154] determine a display width in the display size based on the width ratio indicated by the size ratio information and width information in the projection area information;

[0155] determine a display height in the display size based on the height ratio indicated by the size ratio information and height information in the projection area information.

[0156] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or contains templates that can be used to perform the methods described in the above method embodiments, and specific implementations can refer to the description of the above method embodiments. For brevity, they will not be repeated here.

[0157] Based on the same technical concept, the embodiments of the present disclosure also provide an electronic device. Referring to Figure 5 The structure schematic diagram of the electronic device provided by the embodiments of the present disclosure is shown in FIG. 5, which includes a processor 501, a memory 502, and a bus 503. The memory 502 is used to store execution instructions, including an internal memory 5021 and an external memory 5022. The internal memory 5021 is also called an internal memory, which is used to temporarily store operation data in the processor 501 and exchange data with the external memory 5022 such as a hard disk. The processor 501 exchanges data with the external memory 5022 through the internal memory 5021. When the electronic device 500 is running, the processor 501 and the memory 502 communicate through the bus 503, so that the processor 501 executes the following instructions:

[0158] obtain a projection image collected by a projection camera and a thumbnail corresponding to each three-dimensional model included in the projection image;

[0159] determine region information of an image superimposition region set in the projection image;

[0160] determine superimposition information of the thumbnail corresponding to the three-dimensional model based on projection information of the three-dimensional model in the projection image and the region information, wherein the superimposition information includes a superimposition position and / or a display size on the image superimposition region;

[0161] superimpose the thumbnail corresponding to each three-dimensional model on the image superimposition region of the projection image according to the superimposition information of the thumbnail, to generate a target display image.

[0162] The specific processing procedure of the processor 501 can refer to the description of the above method embodiments, which will not be repeated here.

[0163] In addition, the disclosure embodiments also provide a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the model display method described in the above method embodiments are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0164] The disclosure embodiments also provide a computer program product carrying a program code. The program code includes instructions for executing the steps of the model display method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0165] The computer program product can be implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0166] The disclosure embodiments relate to the field of augmented reality. By obtaining image information of a target object in a real environment, and then using various vision-related algorithms to detect or identify the relevant features, states and attributes of the target object, an AR effect combining virtual and real objects that matches specific applications is obtained. For example, the target object can be related to a face, a limb, a gesture, an action, etc. of a human body, or a marker, a sign, etc. of an object, or a sand table, a display area or a display object, etc. of a venue or a place. The vision-related algorithms can include visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of an object, pose or depth detection of an object, etc. The specific applications can not only include interactive scenarios such as tour, navigation, explanation, reconstruction, virtual effect superimposition display, etc. related to real scenes or objects, but also include interactive scenarios such as makeup beautification, limb beautification, special effect display, virtual model display, etc. related to a person. The detection or identification of the relevant features, states and attributes of the target object can be achieved by a convolutional neural network. The convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system and the device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0168] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0169] In addition, the functional units in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0170] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0171] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A model display method, characterized in that, include: Acquire the projected image captured by the projection camera, and the thumbnail corresponding to each 3D model included in the projected image; The projection camera is a virtual projection camera in the model rendering engine; the thumbnail is an image that can represent the shape of the three-dimensional model; the thumbnail includes the front view and / or top view of the three-dimensional model; Determine the region information of the image overlay area set in the projected image; the image overlay area is a blank area in the projected image; Based on the projection information of the 3D model in the projected image and the region information, the overlay information of the thumbnail corresponding to the 3D model is determined; wherein, the overlay information includes: the overlay position and / or display size on the overlay area of ​​the image; According to the overlay information of the thumbnails, the thumbnail corresponding to each 3D model is overlaid on the image overlay area of ​​the projected image to generate the target display image.

2. The method according to claim 1, characterized in that, Generate a thumbnail for each 3D model, including: Determine the planar view of the three-dimensional model in the target acquisition direction; Based on the plan view, a thumbnail of the 3D model is generated.

3. The method according to claim 1 or 2, characterized in that, When the overlay information includes the overlay position, determining the overlay information of the thumbnail corresponding to the 3D model based on the projection information of the 3D model in the projected image and the region information includes: Based on the region size indicated by the region information and the image size corresponding to the projected image, the size ratio information between the image overlay region and the projected image is determined; Based on the size ratio information and the position information of the three-dimensional model on the projected image indicated by the projection information corresponding to the three-dimensional model, the superposition position of the thumbnail corresponding to the three-dimensional model is determined.

4. The method according to claim 3, characterized in that, Determining the position information of the 3D model on the projected image includes: Based on the projection area information of the 3D model in the projected image, the center point of the projection area corresponding to the 3D model is determined, and the position information of the center point is used as the position information of the 3D model on the projected image; or... Obtain the display position information of the 3D model in the set world coordinate system; Using a determined coordinate system transformation matrix, the display position information is transformed into the view coordinate system corresponding to the projected image, thereby obtaining the position information of the three-dimensional model on the projected image.

5. The method according to claim 3, characterized in that, The method of determining the overlay position of the thumbnail corresponding to the 3D model based on the size ratio information and the projection information corresponding to the 3D model on the projected image, and determining the position information of the 3D model on the projected image, includes: Based on the horizontal coordinate information of the position information of the three-dimensional model on the projected image and the width ratio of the size ratio information, the horizontal coordinate position of the superimposed position is determined. Based on the ordinate information of the position information of the three-dimensional model on the projected image and the height ratio of the size ratio information, the ordinate position of the superimposed position is determined.

6. The method according to claim 1, characterized in that, When the overlay information includes the display size, determining the overlay information of the thumbnail corresponding to the 3D model based on the projection information of the 3D model in the projected image and the region information includes: Based on the region size indicated by the region information and the image size corresponding to the projected image, the size ratio information between the image overlay region and the projected image is determined; Based on the size ratio information and the projection area information of the three-dimensional model in the projected image indicated by the projection information, the display size of the thumbnail corresponding to the three-dimensional model is determined.

7. The method according to claim 6, characterized in that, The determination of the display size of the thumbnail corresponding to the 3D model based on the size ratio information and the projection information indicating the projection area information of the 3D model in the projected image includes: Based on the width ratio indicated by the size ratio information and the width information in the projection area information, the display width in the display size is determined; The display height in the display size is determined based on the height ratio indicated by the size ratio information and the height information in the projection area information.

8. A model display device, characterized in that, include: The acquisition module is used to acquire the projected image captured by the projection camera, and the thumbnail corresponding to each three-dimensional model included in the projected image; The projection camera is a virtual projection camera in the model rendering engine; the thumbnail is an image that can represent the shape of the three-dimensional model; the thumbnail includes the front view and / or top view of the three-dimensional model; The first determining module is used to determine the region information of the image overlay region set in the projected image; the image overlay region is a blank area in the projected image. The second determining module is used to determine the overlay information of the thumbnail corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projected image and the region information; wherein, the overlay information includes: the overlay position and / or display size on the overlay area of ​​the image; The first generation module is used to overlay the thumbnail corresponding to each 3D model onto the image overlay area of ​​the projected image according to the overlay information of the thumbnail, thereby generating a target display image.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the model demonstration method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the model representation method as described in any one of claims 1 to 7.

Images