Model point rendering method and device, storage medium, and electronic device

Abstract

The embodiments of the present disclosure disclose a rendering method and device for model points, a storage medium, and an electronic device, wherein the method comprises: obtaining at least two color panoramas corresponding to at least two known points in a three-dimensional model; determining first depth information of a first color panorama corresponding to a first point among the at least two known points in the direction of a target point, and determining second depth information of a second color panorama corresponding to a second point among the at least two known points in the direction of the target point; determining third depth information of the target point relative to the first point in the three-dimensional model, and fourth depth information of the target point relative to the second point in the three-dimensional model; determining color information corresponding to the target point based on a matching relationship between the first depth information and the third depth information, and / or a matching relationship between the second depth information and the fourth depth information; the embodiment reduces erroneous residual images.

Description

Technical Field

[0001] The present invention relates to a method and device for rendering model points, a storage medium, and an electronic device. Background Art

[0002] When collecting colors in three-dimensional space, the general method is to take multiple panoramic pictures at multiple collection points in a three-dimensional scene; at this time, the color information of the panoramic picture at the collection point is complete, and the corresponding image information can be correctly displayed; however, since the collection points are distributed in isolation, there are some positions between the collection points. The color information has not been collected, resulting in some calculations required to obtain the color information of the positions that have not been collected when rendering the three-dimensional space. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a rendering method and device for model points, a storage medium, and an electronic device.

[0004] According to one aspect of an embodiment of the present disclosure, a method for rendering a model point is provided, comprising:

[0005] Obtain at least two color panoramic images corresponding to at least two known points in the three-dimensional model;

[0006] Determine first depth information of a first color panoramic image corresponding to a first point among the at least two known points in the direction of a target point, and determine second depth information of a second color panoramic image corresponding to a second point among the at least two known points in the direction of the target point; wherein the first point and the second point are adjacent points among the at least two known points;

[0007] Determine third depth information of the target point relative to the first point in the three-dimensional model, and fourth depth information of the target point relative to the second point in the three-dimensional model;

[0008] Color information corresponding to the target point is determined according to a matching relationship between the first depth information and the third depth information, and / or a matching relationship between the second depth information and the fourth depth information.

[0009] Optionally, determining the color information corresponding to the target point according to the matching relationship between the first depth information and the third depth information, and / or the matching relationship between the second depth information and the fourth depth information, includes:

[0010] determining a first difference between the first depth information and the third depth information, and determining color information corresponding to the target point in a first direction relative to the first point and a second direction relative to the second point, or in the second direction, based on a relationship between the first difference and a first set threshold; and / or,

[0011] A second difference between the second depth information and the fourth depth information is determined, and based on a relationship between the second difference and a second set threshold, color information corresponding to the target point is determined in the first direction and the second direction, or in the first direction.

[0012] Optionally, the determining a first difference between the first depth information and the third depth information, based on a relationship between the first difference and a first set threshold, using a first direction of the target point relative to the first point and a second direction of the target point relative to the second point, or using the second direction to determine color information corresponding to the target point, includes:

[0013] determining the first difference;

[0014] In response to the first difference being greater than the first set threshold, determining color information corresponding to the target point in the second direction;

[0015] In response to the first difference being less than or equal to the first set threshold, color information corresponding to the target point is determined in the first direction and the second direction.

[0016] Optionally, the determining a second difference between the second depth information and the fourth depth information, based on a relationship between the second difference and a second set threshold, determining color information corresponding to the target point in the first direction and the second direction, or in the first direction, includes:

[0017] determining the second difference;

[0018] In response to the second difference being greater than the second set threshold, determining color information corresponding to the target point in the first direction;

[0019] In response to the second difference being less than or equal to the second set threshold, color information corresponding to the target point is determined in the first direction and the second direction.

[0020] Optionally, determining the color information corresponding to the target point position based on the first direction and the second direction includes:

[0021] Obtaining first color information from a first color panoramic image corresponding to the first point based on the first direction;

[0022] Obtaining second color information from a second color panoramic image corresponding to the second point based on the second direction;

[0023] Based on the first color information and the second color information, obtaining color information corresponding to the target point by using a synthesis method;

[0024] The determining the color information corresponding to the target point in the first direction includes:

[0025] Obtaining first color information from a first color panoramic image corresponding to the first point based on the first direction, and using the first color information as color information corresponding to the target point;

[0026] The determining the color information corresponding to the target point position in the second direction includes:

[0027] Based on the second direction, second color information is obtained from a second color panoramic image corresponding to the second point, and the second color information is used as the color information corresponding to the target point.

[0028] Optionally, the determining first depth information of the first color panoramic image corresponding to the first point among the at least two known points in the direction of the target point includes:

[0029] In the three-dimensional model, taking the first point as a starting point and the target point as an end point, establishing a first connecting line segment from the first point to the target point;

[0030] Determine a first direction of the target point relative to the first point based on the first point and the first connecting line segment;

[0031] A first pixel point is determined from the first color panoramic image based on the first direction, and the first depth information is determined according to the depth information of the first pixel point.

[0032] Optionally, the determining second depth information of the second color panoramic image corresponding to the second point among the at least two known points in the direction of the target point includes:

[0033] In the three-dimensional model, taking the second point as the starting point and the target point as the ending point, establishing a second connecting line segment from the second point to the target point;

[0034] Determining a second direction of the target point relative to the second point based on the second point and the second connecting line segment;

[0035] A second pixel point is determined from the second color panoramic image based on the second direction, and the second depth information is determined according to the depth information of the second pixel point.

[0036] Optionally, the determining third depth information of the target point in the three-dimensional model, and fourth depth information of the target point relative to the second point in the three-dimensional model, includes:

[0037] Determining the third depth information based on a positional relationship between the target point and the first point in the three-dimensional model;

[0038] The fourth depth information is determined based on a positional relationship between the target point and the second point in the three-dimensional model.

[0039] According to another aspect of an embodiment of the present disclosure, a rendering device for model points is provided, including:

[0040] A panoramic image acquisition module, used to acquire at least two color panoramic images corresponding to at least two known points in the three-dimensional model;

[0041] a depth determination module for determining first depth information of a first color panoramic image corresponding to a first point among the at least two known points in a direction of a target point, and determining second depth information of a second color panoramic image corresponding to a second point among the at least two known points in a direction of the target point; wherein the first point and the second point are adjacent points among the at least two known points;

[0042] A model depth determination module, used to determine third depth information of the target point relative to the first point in the three-dimensional model, and fourth depth information of the target point relative to the second point in the three-dimensional model;

[0043] A target color determination module is used to determine the color information corresponding to the target point according to the matching relationship between the first depth information and the third depth information, and / or the matching relationship between the second depth information and the fourth depth information.

[0044] Optionally, the target color determination module includes:

[0045] A first color determination unit is configured to determine a first difference between the first depth information and the third depth information, and based on a relationship between the first difference and a first set threshold, determine the color information corresponding to the target point in a first direction relative to the first point and a second direction relative to the second point, or in the second direction; and / or,

[0046] A second color determination unit is used to determine a second difference between the second depth information and the fourth depth information, and based on the relationship between the second difference and a second set threshold, determine the color information corresponding to the target point in the first direction and the second direction, or in the first direction.

[0047] Optionally, the first color determination unit is specifically used to determine the first difference; in response to the first difference being greater than the first set threshold, determining the color information corresponding to the target point in the second direction; in response to the first difference being less than or equal to the first set threshold, determining the color information corresponding to the target point in the first direction and the second direction.

[0048] Optionally, the second color determination unit is specifically used to determine the second difference; in response to the second difference being greater than the second set threshold, determining the color information corresponding to the target point in the first direction; in response to the second difference being less than or equal to the second set threshold, determining the color information corresponding to the target point in the first direction and the second direction.

[0049] Optionally, when determining the color information corresponding to the target point in the first direction and the second direction, the first color determination unit or the second color determination unit is used to obtain first color information from a first color panoramic image corresponding to the first point based on the first direction; obtain second color information from a second color panoramic image corresponding to the second point based on the second direction; and obtain the color information corresponding to the target point by using a synthesis method based on the first color information and the second color information;

[0050] When determining the color information corresponding to the target point in the first direction, the second color determination unit is used to obtain first color information from a first color panoramic image corresponding to the first point based on the first direction, and use the first color information as the color information corresponding to the target point;

[0051] When determining the color information corresponding to the target point in the second direction, the first color determination unit is used to obtain second color information from a second color panoramic image corresponding to the second point based on the second direction, and use the second color information as the color information corresponding to the target point.

[0052] Optionally, the image depth determination module is specifically used to establish a first connecting line segment from the first point to the target point in the three-dimensional model, with the first point as the starting point and the target point as the ending point; determine a first direction of the target point relative to the first point based on the first point and the first connecting line segment; determine a first pixel point from the first color panoramic image based on the first direction, and determine the first depth information according to the depth information of the first pixel point.

[0053] Optionally, the image depth determination module is also used to establish a second connecting line segment from the second point to the target point in the three-dimensional model, with the second point as the starting point and the target point as the ending point; determine a second direction of the target point relative to the second point based on the second point and the second connecting line segment; determine a second pixel point from the second color panoramic image based on the second direction, and determine the second depth information according to the depth information of the second pixel point.

[0054] Optionally, the model depth determination module is specifically used to determine the third depth information based on the positional relationship between the target point and the first point in the three-dimensional model; and to determine the fourth depth information based on the positional relationship between the target point and the second point in the three-dimensional model.

[0055] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method for rendering model points described in any of the above embodiments.

[0056] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0057] processor;

[0058] a memory for storing instructions executable by the processor;

[0059] The processor is used to read the executable instructions from the memory and execute the instructions to implement the rendering method of model points described in any one of the above embodiments.

[0060] Based on a rendering method and device for model points, a storage medium, and an electronic device provided in the above-mentioned embodiments of the present disclosure, at least two color panoramas corresponding to at least two known points in a three-dimensional model are obtained; first depth information of a first color panorama corresponding to a first point among the at least two known points in the direction of a target point is determined, and second depth information of a second color panorama corresponding to a second point among the at least two known points in the direction of the target point is determined; wherein the first point and the second point are adjacent points among the at least two known points; third depth information of the target point in the three-dimensional model relative to the first point is determined, and the fourth depth information of the target point relative to the second point in the three-dimensional model; according to a first matching relationship between the first depth information and the third depth information, and a second matching relationship between the second depth information and the fourth depth information, color information corresponding to the target point is determined according to the first matching relationship and / or the second matching relationship; this embodiment determines how to determine the color information of the target point through the matching relationship between the depth information of the target point in the color panoramic image and the depth information in the model, thereby avoiding errors caused by determining the color information in combination with the color panoramic image when the target point is blocked in the color panoramic image corresponding to a known point.

[0061] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0063] Figure 1 It is a flowchart of a method for rendering model points provided by an exemplary embodiment of the present disclosure.

[0064] Figure 2 It is a schematic diagram of known points in an optional example of a method for rendering model points provided by an exemplary embodiment of the present disclosure.

[0065] Figure 3 This disclosure Figure 1 A schematic flow chart of step 104 in the illustrated embodiment.

[0066] Figure 4 This disclosure Figure 1 Another flowchart diagram of step 104 in the embodiment shown.

[0067] Figure 5 It is a schematic diagram of depth information determination in an optional example of a rendering method for model points provided by an exemplary embodiment of the present disclosure.

[0068] Figure 6 It is a schematic diagram of the structure of a rendering device for model points provided by an exemplary embodiment of the present disclosure.

[0069] Figure 7 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0071] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0072] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0073] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0074] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0075] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the previous and next associated objects are in an "or" relationship. The data referred to in this disclosure may include unstructured data such as text, images, and videos, and may also be structured data.

[0076] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0077] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0078] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0079] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0080] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0081] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0082] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0083] Application Overview

[0084] In the process of implementing the present disclosure, the inventors discovered that in the prior art, when rendering a three-dimensional space, the transition point between acquisition points is determined by synthesizing the colors corresponding to the two acquisition points at the transition point, but the prior art has at least the following problem: when the transition point is blocked in the panoramic image corresponding to one acquisition point, it cannot be rendered correctly.

[0085] Exemplary Methods

[0086] Figure 1 FIG. 1 is a flow chart of a method for rendering model points provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:

[0087] Step 102: Obtain at least two color panoramic images corresponding to at least two known points in the three-dimensional model.

[0088] Among them, the three-dimensional model can be a model collected in any three-dimensional space. Optionally, the method for obtaining the three-dimensional model can include but is not limited to: collecting images of the three-dimensional space through a depth camera that can obtain depth information, and constructing a three-dimensional model with the obtained images and depth; collecting at least two color panoramas from at least two points with a panoramic camera, and using a pre-trained neural network model for predicting image depth to perform depth prediction on the color panorama to obtain depth information of each pixel in the color panorama, and obtaining a point cloud of the three-dimensional space with the color panorama with depth information, converting the point cloud into a model (mesh), and obtaining a three-dimensional model, etc. Optionally, the known point in this embodiment can be the position point of the camera device that collects the color panorama.

[0089] Step 104, determining first depth information of a first color panoramic image corresponding to a first point among at least two known points in the direction of a target point, and determining second depth information of a second color panoramic image corresponding to a second point among at least two known points in the direction of the target point.

[0090] The first point and the second point are adjacent points among at least two known points. The target point can be a point on the three-dimensional model corresponding to any position between the first point and the second point. For example, in one example, the three-dimensional model is as follows: Figure 2 In the floor plan shown in the figure (the relationship between points is only shown from a bird's-eye view), the first point is point A in the figure, the second point is point B in the figure, point C in the figure is the transition point from point A to point B, and the target point can be the point on the model corresponding to point C as the observation point; by rendering multiple model points corresponding to multiple transition points between point A and point B, a smooth transition in color of the three-dimensional model viewed when moving from point A to point B can be achieved.

[0091] Step 106: Determine third depth information of the target point relative to the first point in the three-dimensional model, and fourth depth information of the target point relative to the second point in the three-dimensional model.

[0092] In this embodiment, since the target point is a point on the three-dimensional model, the third depth information can be determined based on the positional relationship between the target point and the first point in the three-dimensional model, and the fourth depth information can be determined based on the positional relationship between the target point and the second point in the three-dimensional model.

[0093] Step 108: Determine color information corresponding to the target point according to the matching relationship between the first depth information and the third depth information, and / or the matching relationship between the second depth information and the fourth depth information.

[0094] In this embodiment, a plurality of schemes for determining the color information of the target point are proposed. By selecting different schemes in different situations, more accurate color information is determined for the target point.

[0095] The above-mentioned embodiment of the present disclosure provides a rendering method for a model point, which obtains at least two color panoramas corresponding to at least two known points in a three-dimensional model; determines first depth information of a first color panorama corresponding to a first point among the at least two known points in the direction of a target point, and determines second depth information of a second color panorama corresponding to a second point among the at least two known points in the direction of the target point; wherein the first point and the second point are adjacent points among the at least two known points; determines third depth information of the target point in the three-dimensional model relative to the first point, and fourth depth information of the target point in the three-dimensional model relative to the second point; determines color information corresponding to the target point according to a matching relationship between the first depth information and the third depth information, and / or a matching relationship between the second depth information and the fourth depth information; in this embodiment, how to determine the color information of the target point is determined by matching the depth information of the target point in the color panorama with the depth information in the model, thereby avoiding errors caused by determining the color information in combination with the color panorama when the target point is blocked in the color panorama corresponding to a certain known point.

[0096] In some optional embodiments, step 108 in the above embodiment may include:

[0097] Determine a first difference between the first depth information and the third depth information, and determine color information corresponding to the target point based on a first direction of the target point relative to the first point and a second direction of the target point relative to the second point, or based on the second direction, based on a relationship between the first difference and a first set threshold; and / or,

[0098] A second difference between the second depth information and the fourth depth information is determined, and based on a relationship between the second difference and a second set threshold, color information corresponding to the target point is determined in the first direction and the second direction, or in the first direction.

[0099] In this embodiment, in order to ensure the accuracy of the relationship between the first difference, the second difference and their corresponding set thresholds, the absolute value of the first difference can be compared with the first set threshold, and the absolute value of the second difference can be compared with the second set threshold; optionally, the relationship between the first difference and the first set threshold, and the relationship between the second difference and the second set threshold can be judged separately, or the relationship between the first difference and the first set threshold, and the relationship between the second difference and the second set threshold can be judged successively; for example, the relationship between the first difference and the first set threshold is used to determine the first direction and the second direction (for example, when the absolute value of the first difference is less than or equal to the first set threshold), or the color information corresponding to the target point is determined in the second direction (for example, when the absolute value of the first difference is greater than the first set threshold). threshold); or, determining the color information corresponding to the target point in the first direction and the second direction according to the relationship between the second difference and the second set threshold (for example, when the absolute value of the second difference is less than or equal to the second set threshold), or determining the color information corresponding to the target point in the first direction (for example, when the absolute value of the second difference is greater than the second set threshold); for another example, determining the color information corresponding to the target point in the second direction according to the relationship between the first difference and the first set threshold (for example, when the absolute value of the first difference is greater than the first set threshold), or, determining the color information corresponding to the target point in the first direction according to the first direction and the second direction (for example, when the absolute value of the first difference is less than or equal to the first set threshold), or according to the relationship between the second difference and the second set threshold (for example, when the absolute value of the first difference is less than or equal to the first set threshold).

[0100] Optionally, determining a first difference between the first depth information and the third depth information, and determining color information corresponding to the target point in the first direction and the second direction, or in the second direction, based on a relationship between the first difference and a first set threshold, includes:

[0101] determining a first difference;

[0102] In response to the first difference being greater than a first set threshold, determining color information corresponding to the target point in a second direction;

[0103] In response to the first difference being less than or equal to a first set threshold, color information corresponding to the target point is determined in the first direction and the second direction.

[0104] In this embodiment, the first difference may be an absolute value of a difference between the first depth information and the third depth information, and the absolute value is compared with a first set threshold value. When the first difference is greater than the first set threshold value, it indicates that the depth information of the target point in the model is significantly different from the depth information in the first color panorama, and there may be occlusion in the first color panorama. At this time, if the color information corresponding to the target point is determined by the first direction and the second direction, residual images that should not appear will appear in the color information of the target point, resulting in rendering errors of the three-dimensional model. Therefore, in this embodiment, when the first difference is greater than the first set threshold value, the color information of the target point is determined by the second depth information corresponding to the second point. When the first difference is less than or equal to the first set threshold value, it indicates that the target point is not blocked in the first color panorama, and it is more accurate to determine the color information corresponding to the target point by the first direction and the second direction.

[0105] Optionally, determining a second difference between the second depth information and the fourth depth information, and determining color information corresponding to the target point in the first direction and the second direction, or in the first direction, based on a relationship between the second difference and a second set threshold, includes:

[0106] determining a second difference;

[0107] In response to the second difference being greater than a second set threshold, determining color information corresponding to the target point in the first direction and the second direction;

[0108] In response to the second difference being less than or equal to the second set threshold, color information corresponding to the target point is determined in the first direction.

[0109] In this embodiment, the second difference may be an absolute value of a difference between the second depth information and the fourth depth information, and the absolute value is compared with the second set threshold value. When the second difference is greater than the second set threshold value, it indicates that the depth information of the target point in the model is significantly different from the depth information in the second color panorama, and there may be occlusion in the second color panorama. At this time, if the color information corresponding to the target point is determined by the first direction and the second direction, residual images that should not appear will appear in the color information of the target point, resulting in rendering errors of the three-dimensional model. Therefore, in this embodiment, when the second difference is greater than the second set threshold value, the color information of the target point is determined by the first direction corresponding to the first point. When the second difference is less than or equal to the second set threshold value, it indicates that the target point is not blocked in the second color panorama, and it is more accurate to determine the color information corresponding to the target point by the first direction and the second direction.

[0110] Through the above-mentioned technical features, this embodiment solves the technical problem of how to more accurately determine the color information of the target point in the model when the target point is blocked, and avoids the problem of residual images in the rendering results caused by only comprehensively determining the color information based on two depth information in the prior art.

[0111] Optionally, determining color information corresponding to the target point in the first direction and the second direction includes:

[0112] First color information is obtained from a first color panoramic image corresponding to a first point based on a first direction; second color information is obtained from a second color panoramic image corresponding to a second point based on a second direction.

[0113] Optionally, after determining the direction, a corresponding position point can be determined in the first color panorama based on the first direction, and the color information of the position point is used as the first color information. Correspondingly, a corresponding position point can be determined in the second color panorama based on the second direction, and the color information of the position point is used as the second color information.

[0114] Based on the first color information and the second color information, a synthesis method is used to obtain color information corresponding to the target point.

[0115] In this embodiment, the method of obtaining the color information corresponding to the target point based on the first color information and the second color information can be implemented based on the color synthesis method in the prior art. For example, transparent synthesis (also known as alpha synthesis, alpha synthesis or alpha blending) is the process of combining a semi-transparent foreground color with a background color to obtain a mixed new color. The transparency of the foreground color is not limited, ranging from completely transparent to completely opaque. In this embodiment, the first color information and the second color information are used as the foreground color and background color, and both are non-transparent. Through transparent synthesis, the color information of the target point can be obtained.

[0116] Optionally, determining color information corresponding to the target point in the first direction includes:

[0117] Obtaining first color information from a first color panoramic image corresponding to a first point based on a first direction, and using the first color information as color information corresponding to a target point;

[0118] Determine the color information corresponding to the target point in the second direction, including:

[0119] Based on the second direction, second color information is obtained from a second color panoramic image corresponding to the second point, and the second color information is used as color information corresponding to the target point.

[0120] like Figure 3 As shown in the above Figure 1Based on the illustrated embodiment, the process of determining the first depth information in step 104 may include the following steps:

[0121] Step 1041: In the three-dimensional model, a first connecting line segment is established from the first point to the target point with the first point as the starting point and the target point as the ending point.

[0122] Step 1042: Determine a first direction of the target point relative to the first point based on the first point and the first connecting line segment.

[0123] Step 1043: determine a first pixel point from the first color panoramic image based on the first direction, and determine first depth information according to the depth information of the first pixel point.

[0124] In this embodiment, the first direction of the target point relative to the first point is first determined through the positional relationship between the first point and the target point in the three-dimensional model, and then the first pixel point corresponding to the first direction is searched in the first color panoramic image in the first direction. In the color panoramic image, each pixel point has corresponding depth information (and color information), and the depth information of the pixel point is used as the first depth information.

[0125] like Figure 4 As shown in the above Figure 1 Based on the illustrated embodiment, the process of determining the second depth information in step 104 may include the following steps:

[0126] Step 1044: In the three-dimensional model, a second connecting line segment is established from the second point to the target point with the second point as the starting point and the target point as the ending point.

[0127] Step 1045: Determine a second direction of the target point relative to the second point based on the second point and the second connecting line segment.

[0128] Step 1046: determine a second pixel point from the second color panoramic image based on the second direction, and determine second depth information according to the depth information of the second pixel point.

[0129] In this embodiment, the second direction of the target point relative to the second point is first determined through the positional relationship between the second point and the target point in the three-dimensional model, and then the second pixel corresponding to the second direction is searched and determined in the second color panorama in the second direction, and each pixel in the color panorama has corresponding color information. A pre-trained neural network model for predicting image depth is used to perform depth prediction on the color panorama, and the depth information of each pixel in the color panorama can be obtained, and the depth information of the pixel is used as the second depth information.

[0130] In some optional embodiments, step 106 may include:

[0131] Determining third depth information based on a positional relationship between the target point and the first point in the three-dimensional model;

[0132] Based on the positional relationship between the target point and the second point in the three-dimensional model, fourth depth information is determined.

[0133] Alternatively, in an optional example, such as Figure 5 As shown (the figure only shows the point relationship from a top-down perspective), the third depth information between the target point K and the first point A in the three-dimensional model can be determined by calculating the distance between the two in the three-dimensional model. For example, the line segment AK in the figure can represent the third depth information, and because the target point K is blocked when viewed from point A, the line segment AM can represent the corresponding first depth information; similarly, the fourth depth information between the target point K and the second point B in the three-dimensional model can be determined by calculating the distance between the two in the three-dimensional model; after determining the third depth information and the fourth depth information, the efficiency of comparing with the depth information in the color panorama can be improved, thereby improving the efficiency of determining the color information of the target point.

[0134] The rendering method of model points provided in the embodiments of the present disclosure is applied to color rendering of three-dimensional models to solve the problem of transition of color rendering from one known point to another known point. Specifically, on the premise that the color panorama corresponding to two adjacent known points is known, based on the rendering method of model points provided in the above embodiments, the color information of multiple model points on the model corresponding to the continuous transition points between the two known points is determined, and the three-dimensional model is rendered based on the color information of the multiple model points, thereby achieving a smooth transition from one known point to another known point.

[0135] Any model point rendering method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any model point rendering method provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any model point rendering method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0136] Exemplary Devices

[0137] Figure 6 FIG. 1 is a schematic diagram of a structure of a rendering device for model points provided by an exemplary embodiment of the present disclosure. Figure 6 As shown, the device provided in this embodiment includes:

[0138] The panoramic image acquisition module 61 is used to acquire at least two color panoramic images corresponding to at least two known points in the three-dimensional model.

[0139] The image depth determination module 62 is used to determine the first depth information of the first color panoramic image corresponding to the first point among at least two known points in the direction of the target point, and to determine the second depth information of the second color panoramic image corresponding to the second point among at least two known points in the direction of the target point.

[0140] The first point and the second point are adjacent points among at least two known points.

[0141] The model depth determination module 63 is used to determine third depth information of the target point relative to the first point in the three-dimensional model, and fourth depth information of the target point relative to the second point in the three-dimensional model.

[0142] The target color determination module 64 is used to determine the color information corresponding to the target point according to the matching relationship between the first depth information and the third depth information and / or the matching relationship between the second depth information and the fourth depth information.

[0143] The above-mentioned embodiment of the present disclosure provides a rendering device for model points, which obtains at least two color panoramas corresponding to at least two known points in a three-dimensional model; determines first depth information of a first color panorama corresponding to a first point among the at least two known points in the direction of a target point, and determines second depth information of a second color panorama corresponding to a second point among the at least two known points in the direction of the target point; wherein the first point and the second point are adjacent points among the at least two known points; determines third depth information of the target point in the three-dimensional model relative to the first point, and fourth depth information of the target point in the three-dimensional model relative to the second point; determines color information corresponding to the target point according to a matching relationship between the first depth information and the third depth information, and / or a matching relationship between the second depth information and the fourth depth information; in this embodiment, how to determine the color information of the target point is determined by matching the depth information of the target point in the color panorama with the depth information in the model, thereby avoiding errors caused by determining the color information in combination with the color panorama when the target point is blocked in the color panorama corresponding to a certain known point.

[0144] Optionally, the target color determination module 64 includes:

[0145] A first color determination unit is used to determine a first difference between the first depth information and the third depth information, and based on a relationship between the first difference and a first set threshold, determine color information corresponding to the target point in a first direction relative to the first point and a second direction relative to the second point, or in the second direction; and / or,

[0146] The second color determination unit is used to determine a second difference between the second depth information and the fourth depth information, and based on the relationship between the second difference and the second set threshold, determine the color information corresponding to the target point in the first direction and the second direction, or in the first direction.

[0147] Optionally, the first color determination unit is specifically used to determine a first difference; in response to the first difference being greater than a first set threshold, determine the color information corresponding to the target point in the second direction; in response to the first difference being less than or equal to the first set threshold, determine the color information corresponding to the target point in the first direction and the second direction.

[0148] Optionally, the second color determination unit is specifically used to determine the second difference; in response to the second difference being greater than a second set threshold, determine the color information corresponding to the target point in the first direction; in response to the second difference being less than or equal to the second set threshold, determine the color information corresponding to the target point in the first direction and the second direction.

[0149] Optionally, when the first color determination unit or the second color determination unit determines the color information corresponding to the target point in the first direction and the second direction, it is used to obtain the first color information from the first color panoramic image corresponding to the first point based on the first direction; obtain the second color information from the second color panoramic image corresponding to the second point based on the second direction; and obtain the color information corresponding to the target point by using a synthesis method based on the first color information and the second color information;

[0150] When the second color determination unit determines the color information corresponding to the target point in the first direction, it is used to obtain the first color information from the first color panoramic image corresponding to the first point based on the first direction, and use the first color information as the color information corresponding to the target point;

[0151] When the first color determination unit determines the color information corresponding to the target point in the second direction, it is used to obtain second color information from the second color panoramic image corresponding to the second point based on the second direction, and use the second color information as the color information corresponding to the target point.

[0152] Optionally, the image depth determination module 62 is specifically used to establish a first connecting line segment from the first point to the target point in the three-dimensional model, with the first point as the starting point and the target point as the ending point; determine a first direction of the target point relative to the first point based on the first point and the first connecting line segment; determine a first pixel point from the first color panoramic image based on the first direction, and determine first depth information according to the depth information of the first pixel point.

[0153] Optionally, the image depth determination module 62 is also used to establish a second connecting line segment from the second point to the target point in the three-dimensional model, with the second point as the starting point and the target point as the ending point; determine a second direction of the target point relative to the second point based on the second point and the second connecting line segment; determine a second pixel point from the second color panoramic image based on the second direction, and determine second depth information according to the depth information of the second pixel point.

[0154] Optionally, the model depth determination module 63 is specifically used to determine the third depth information based on the positional relationship between the target point and the first point in the three-dimensional model; and determine the fourth depth information based on the positional relationship between the target point and the second point in the three-dimensional model.

[0155] Exemplary Electronic Devices

[0156] Below, reference Figure 7 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device 100 and the second device 200, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0157] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0158] like Figure 7 As shown, the electronic device 70 includes one or more processors 71 and a memory 72 .

[0159] The processor 71 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 70 to perform desired functions.

[0160] The memory 72 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 71 may run the program instructions to implement the rendering method of the model points of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0161] In one example, the electronic device 70 may further include: an input device 73 and an output device 74, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0162] For example, when the electronic device is the first device 100 or the second device 200, the input device 73 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a stand-alone device, the input device 73 may be a communication network connector, for receiving collected input signals from the first device 100 and the second device 200.

[0163] In addition, the input device 73 may also include, for example, a keyboard, a mouse, and the like.

[0164] The output device 74 can output various information to the outside, including the determined distance information, direction information, etc. The output device 74 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0165] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device 70 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 70 may also include any other appropriate components.

[0166] Exemplary computer program products and computer-readable storage media

[0167] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the rendering method of model points according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0168] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0169] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method for rendering model points according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0170] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0171] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0172] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0174] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0175] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0176] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0177] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for rendering model points, characterized in that: include: Acquire at least two color panoramas corresponding to at least two known points in the three-dimensional model, wherein the known points are positions of a camera device for capturing the color panoramas; Determine first depth information of a first color panoramic image corresponding to a first point among the at least two known points in the direction of a target point, and determine second depth information of a second color panoramic image corresponding to a second point among the at least two known points in the direction of the target point; wherein the first point and the second point are adjacent points among the at least two known points, and the target point is a point on a three-dimensional model corresponding to any position between the first point and the second point; Determine third depth information of the target point relative to the first point in the three-dimensional model, and fourth depth information of the target point relative to the second point in the three-dimensional model; Determine color information corresponding to the target point according to a matching relationship between the first depth information and the third depth information, and / or a matching relationship between the second depth information and the fourth depth information; Wherein, determining the color information corresponding to the target point according to the matching relationship between the first depth information and the third depth information, and / or the matching relationship between the second depth information and the fourth depth information, includes: Determine a first difference between the first depth information and the third depth information, and based on the relationship between the first difference and a first set threshold, determine the color information corresponding to the target point in a first direction relative to the first point and a second direction relative to the second point, or in the second direction; and / or, determine a second difference between the second depth information and the fourth depth information, and based on the relationship between the second difference and a second set threshold, determine the color information corresponding to the target point in the first direction and the second direction, or in the first direction; Determining the color information corresponding to the target point in the first direction and the second direction includes: synthesizing the color information corresponding to the target point according to the first color information and the second color information, wherein the first color information is obtained from the first color panorama based on the first direction, and the second color information is obtained from the second color panorama based on the second direction; Determining the color information corresponding to the target point in the first direction includes: using the first color information as the color information corresponding to the target point; Determining the color information corresponding to the target point in the second direction includes: using the second color information as the color information corresponding to the target point.

2. The method according to claim 1, characterized in that: The determining of a first difference between the first depth information and the third depth information, based on a relationship between the first difference and a first set threshold, using a first direction of the target point relative to the first point and a second direction of the target point relative to the second point, or using the second direction to determine color information corresponding to the target point, includes: determining the first difference; In response to the first difference being greater than the first set threshold, determining color information corresponding to the target point in the second direction; In response to the first difference being less than or equal to the first set threshold, color information corresponding to the target point is determined in the first direction and the second direction.

3. The method according to claim 1 or 2, characterized in that: The determining of a second difference between the second depth information and the fourth depth information, based on a relationship between the second difference and a second set threshold, determining color information corresponding to the target point in the first direction and the second direction, or in the first direction, includes: determining the second difference; In response to the second difference being greater than the second set threshold, determining color information corresponding to the target point in the first direction; In response to the second difference being less than or equal to the second set threshold, color information corresponding to the target point is determined in the first direction and the second direction.

4. The method according to claim 1 or 2, characterized in that: The determining of first depth information of a first color panoramic image corresponding to a first point among the at least two known points in a direction of a target point includes: In the three-dimensional model, taking the first point as a starting point and the target point as an end point, establishing a first connecting line segment from the first point to the target point; Determine a first direction of the target point relative to the first point based on the first point and the first connecting line segment; A first pixel point is determined from the first color panoramic image based on the first direction, and the first depth information is determined according to the depth information of the first pixel point.

5. The method according to claim 1 or 2, characterized in that: The step of determining second depth information of the second color panoramic image corresponding to the second point among the at least two known points in the direction of the target point includes: In the three-dimensional model, taking the second point as the starting point and the target point as the ending point, establishing a second connecting line segment from the second point to the target point; Determining a second direction of the target point relative to the second point based on the second point and the second connecting line segment; A second pixel point is determined from the second color panoramic image based on the second direction, and the second depth information is determined according to the depth information of the second pixel point.

6. The method according to claim 1 or 2, characterized in that: The determining of the third depth information of the target point relative to the first point in the three-dimensional model, and the fourth depth information of the target point relative to the second point in the three-dimensional model, includes: Determining the third depth information based on a positional relationship between the target point and the first point in the three-dimensional model; The fourth depth information is determined based on a positional relationship between the target point and the second point in the three-dimensional model.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the rendering method of model points described in any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the model point rendering method described in any one of claims 1-6 above.

Images