Three-dimensional information determination method and apparatus, device, storage medium and program product

Abstract

A three-dimensional information determination method and apparatus, a device and a storage medium, relating to the technical field of image processing. The method can be applied to scenarios such as cloud technology, intelligent transportation and assisted driving. The method comprises: acquiring an image obtained by photographing the outer surface of a photographed object, wherein n protruding three-dimensional units are arranged on the outer surface of the photographed object, the three-dimensional units are entity units having a three-dimensional structure, and n is a positive integer; processing the image to obtain box information of at least one three-dimensional unit, wherein the box information is used for positioning the first surface and the second surface of the three-dimensional unit in the image, the first surface is parallel to the second surface, and the first surface or the second surface is in the outer surface; and determining three-dimensional information of each three-dimensional unit on the basis of the box information of the three-dimensional unit, wherein the three-dimensional information is used for representing the position of the three-dimensional unit in the image and the degree of protrusion of the three-dimensional unit in the direction perpendicular to the outer surface. The method facilitates reduction of the difficulty in acquiring the three-dimensional information of the three-dimensional units.

Description

Method, device, equipment, storage medium and program product for determining three-dimensional information

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 24, 2023, with application number 2023113786308 and application name “Method, device, equipment and storage medium for determining three-dimensional information”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to the determination of three-dimensional information. Background Art

[0003] By detecting and identifying the stereo unit in the image, information related to the position and size of the stereo unit can be obtained, which facilitates reconstruction of the stereo unit.

[0004] Related technologies use deep learning models to determine the 3D information of stereoscopic elements in an image based on the image and the intrinsic parameters of the camera that captured it. Because different cameras have independent intrinsic parameters, this method can only be executed if the camera intrinsic parameters corresponding to the image are available. This method for determining 3D information has limited adaptability to different scenes and requires further improvement.

[0005] Summary of the Invention

[0006] This application provides a method, apparatus, device, storage medium, and program product for determining three-dimensional information, which can identify stereo units in an image without obtaining other external reference information (such as camera intrinsic parameters). The technical solution is as follows:

[0007] According to one aspect of an embodiment of the present application, a method for determining three-dimensional information is provided, the method comprising:

[0008] Acquire an image obtained by photographing the outer surface of the photographed object; wherein the outer surface of the photographed object has n protruding three-dimensional units, where the three-dimensional units refer to solid units with a three-dimensional structure, and n is a positive integer;

[0009] Processing the image to obtain frame information of at least one of the three-dimensional units; wherein the frame information is used to locate a first surface and a second surface of the three-dimensional unit in the image, the first surface is parallel to the second surface, and the first surface or the second surface is within the outer surface;

[0010] The three-dimensional information of the three-dimensional unit is determined according to the frame information of the three-dimensional unit; wherein the three-dimensional information is used to represent the position of the three-dimensional unit in the image and the degree of protrusion of the three-dimensional unit in a direction perpendicular to the outer surface.

[0011] According to one aspect of an embodiment of the present application, a device for determining three-dimensional information is provided, the device comprising:

[0012] An image acquisition module, configured to acquire an image of the outer surface of the photographed object; wherein the outer surface of the photographed object has n protruding three-dimensional units, wherein the three-dimensional units are solid units having a three-dimensional structure, and n is a positive integer;

[0013] a frame determination module, configured to process the image to obtain frame information of at least one of the three-dimensional units; wherein the frame information is used to locate a first surface and a second surface of the three-dimensional unit in the image, wherein the first surface is parallel to the second surface, and the first surface or the second surface is within the outer surface;

[0014] A three-dimensional determination module is used to determine the three-dimensional information of the three-dimensional unit based on the frame information of the three-dimensional unit; wherein the three-dimensional information is used to characterize the position of the three-dimensional unit in the image and the degree of protrusion of the three-dimensional unit in a direction perpendicular to the outer surface.

[0015] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method for determining three-dimensional information as described above.

[0016] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the three-dimensional information determination method as described above.

[0017] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium, and a processor reads and executes the computer program from the computer-readable storage medium to implement the method for determining three-dimensional information as described above.

[0018] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0019] The method provided in the present application only needs to use the image obtained by photographing the object to be photographed when determining the three-dimensional information of the stereo unit, and by processing the image, obtains the frame information for locating the position of the stereo unit in the image. Subsequently, based on the imaging regularity of multiple stereo units between the outer surfaces of the photographed object, the three-dimensional information of the stereo unit can be obtained only through the frame information of the stereo unit. Compared with the related art that requires the use of external reference information such as camera intrinsic parameters to determine the depth information of the stereo unit, this method does not need to obtain other external reference information (such as camera intrinsic parameters) when determining the three-dimensional information of the stereo unit. The number of reference information needed to determine the three-dimensional information of the stereo unit is reduced, and the three-dimensional information of the stereo unit is determined only by relying on the image. This method helps to improve the adaptability of the three-dimensional information determination method to different scenes, and at the same time, helps to reduce the difficulty of obtaining the three-dimensional information of the stereo unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 is a schematic diagram of an implementation environment of a solution provided by an exemplary embodiment of the present application;

[0021] FIG2 is a schematic diagram of the vanishing point of an image during perspective projection;

[0022] FIG3 is a schematic diagram of perspective distortion occurring during perspective imaging;

[0023] FIG4 is a schematic diagram of identifying a stereo unit from an image based on image segmentation in the related art;

[0024] FIG5 is a schematic diagram of the inventive concept of the three-dimensional information determination method provided by the present application;

[0025] FIG6 is a schematic diagram showing the principle of determining three-dimensional information provided by an exemplary embodiment of the present application;

[0026] FIG7 is a flow chart of a method for determining three-dimensional information provided by an exemplary embodiment of the present application;

[0027] FIG8 is a schematic diagram of a surface of a three-dimensional unit provided by an exemplary embodiment of the present application;

[0028] FIG9 is a schematic diagram of a second surface position determination process provided by an exemplary embodiment of the present application;

[0029] FIG10 is a schematic diagram of a vanishing point determination process provided by an exemplary embodiment of the present application;

[0030] FIG11 is a schematic diagram showing a comparison of border information and position information provided by an exemplary embodiment of the present application;

[0031] FIG12 is a schematic diagram of a corrected stereoscopic unit provided by an exemplary embodiment of the present application;

[0032] FIG13 is a schematic diagram of image cropping correction provided by an exemplary embodiment of the present application;

[0033] FIG14 is a schematic diagram of a process for determining border information provided by an exemplary embodiment of the present application;

[0034] FIG15 is a schematic diagram of a method for determining three-dimensional information provided by an exemplary embodiment of the present application;

[0035] FIG16 is a block diagram of an apparatus for determining three-dimensional information provided by an exemplary embodiment of the present application;

[0036] FIG17 is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0038] Street view images: These are panoramic images of streets, cities, rural areas, and other areas captured by devices such as vehicle-mounted cameras and handheld cameras. The images in the claims may be street view images. Street view images are used to display the corresponding streets, buildings, traffic signs, pedestrians, and other content. Street view images have a wide range of applications in electronic map production, urban planning, tourism navigation, educational research, and other fields.

[0039] Architectural unit: A building unit is a design or decoration on a building's exterior walls or facades, and is an important component of a building's appearance. The three-dimensional units referred to in the claims of this application may be architectural units. Types of architectural units include, but are not limited to, at least one of the following: windows, balconies, and doors.

[0040] The stereo bounding box is used to characterize the edge frame of the stereo unit in the image. The stereo bounding box can be called a three-dimensional bounding box (3D Box). The stereo bounding box includes the inner surface of the stereo unit and the outer surface of the stereo unit. In the application scenario of the present application, the inner surface of the stereo unit is the surface in the stereo unit that is in contact with the photographed object, and the outer surface of the stereo unit is parallel to the inner surface of the stereo unit, that is, there can be a certain distance between the outer surface of the stereo unit and the outer surface of the photographed object. Due to the perspective imaging of the camera, the image of the outer surface of the stereo unit in the image is usually larger than the image of the inner surface of the stereo unit in the image.

[0041] Vanishing point: In perspective projection, the point where a bundle of parallel lines in three-dimensional space that are not parallel to the projection plane converge in the image. This point is also called the vanishing point.

[0042] Camera intrinsic parameters: These are the internal parameters of the camera. They include at least one of the following: focal length, pixel size, and principal point position.

[0043] Perspective distortion refers to the relationship between the size of a subject in an image and its distance from the camera. Because cameras mimic the human eye's visual perspective, images produced by cameras follow the principle of larger objects appearing closer and smaller objects appearing farther away. Objects closer to the camera appear larger in the image, while objects farther away appear smaller.

[0044] FIG1 is a schematic diagram of an implementation environment of a solution provided by an exemplary embodiment of the present application, which may include: a computer device 10 , a terminal device 20 , and a server 30 .

[0045] The computer device 10 includes, but is not limited to, a personal computer (PC), a tablet computer, a mobile phone, a wearable device, a smart home appliance, an in-vehicle terminal, and other electronic devices with computing and storage capabilities. In some embodiments, the computer device is used to acquire an image and predict three-dimensional information of a stereo unit included in the image.

[0046] The terminal device 20 can be an electronic device such as a personal computer, tablet computer, mobile phone, wearable device, smart home appliance, vehicle-mounted terminal, aircraft, or the like. The terminal device 20 is equipped with a camera, or image transmission occurs between the terminal device 20 and the camera. The terminal device 20 uses the camera to capture an image of a photographed object. The photographed plane of the photographed object includes multiple stereoscopic elements, and the image includes the images of the stereoscopic elements.

[0047] The terminal device 20 runs a client for a target application. The target application is used to provide a 3D information acquisition function for stereo units. For example, the terminal device 20 uses the target application to acquire 3D information of stereo units included in an image, thereby enabling tasks such as city restoration, streetscape reconstruction, autonomous driving, and vehicle navigation based on the 3D information of the stereo units.

[0048] The server 30 is used to provide background services for the client of the target application in the terminal device 20. For example, the server 30 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited to these. The server 30 has at least a data receiving function.

[0049] Optionally, the function of determining the three-dimensional information of the computer device 10 may be implemented by the server 30 (the computer device 10 and the server 30 are the same device) or the computer device 10 and the server 30 are two independent devices.

[0050] In one example, after receiving an image uploaded by terminal device 20 via a target application, server 30 transmits the image to computer device 10. Computer device 10 receives the image from server 30 and processes it to determine the three-dimensional information of the stereo unit. Computer device 10 then sends the three-dimensional information of the stereo unit to server 30, which then forwards it to terminal device 20.

[0051] In another example, the computer device 10 obtains an image through a network and determines the three-dimensional information of the stereo unit from the image. The computer device 10 can then use the three-dimensional information of the stereo unit to perform three-dimensional reconstruction of the photographed object.

[0052] Figure 2 is a schematic diagram of the vanishing point of an image during perspective projection. As shown in Figure 2, due to the influence of perspective imaging, the images of a bundle of parallel lines that are not parallel to the projection plane (including lines 210 and 212) in the image intersect at vanishing point 201. The images of a bundle of parallel lines that are parallel to the projection plane (including lines 220 and 222) in the image remain parallel.

[0053] Figure 3 is a schematic diagram of perspective distortion during perspective imaging. As shown in Figure 3, subjects 1 and 2 are the same size. The distance between subject 1 and the viewpoint is greater than the distance between subject 2 and the viewpoint. The size of subject 1 in the imaging plane (where the imaging plane is identical to the image captured, the image is on the left side of the camera in the figure, the imaging plane is on the right side, and the distance between the image and the viewpoint is equal to the distance between the imaging plane and the viewpoint) is larger than the size of subject 2 in the imaging plane. In other words, perspective distortion causes the subjects in the image to appear larger when closer and smaller when farther away.

[0054] Related technologies for detecting the shape of stereo units based on images include the following: One method is to establish shape grammar rules based on the arrangement of stereo units on the outer surface of the photographed object; then, based on the shape grammar rules, the image is processed to predict the geometric shape of the stereo units (e.g., the shape of the outer surface).

[0055] Another method is to segment the image using a deep learning model to determine the location of the stereo unit in the image, and then determine the geometric shape of the stereo unit. Figure 4 is a schematic diagram of the related art for identifying stereo units based on image segmentation. As shown in Figure 4, the related art inputs the image into the deep learning model, and the deep learning model encoder encodes the image to obtain encoded information. The decoder in the deep learning model decodes the encoded information to obtain the image segmentation result. Each fill pattern in the segmentation result corresponds to a stereo unit.

[0056] Because the image obtained by photographing the subject is a two-dimensional image, it cannot fully reflect the various surfaces of the stereoscopic unit. In other words, at least one surface of the stereoscopic unit (at least one surface parallel to the normal of the outer surface of the photographed object) cannot be fully displayed in the image. This makes it impossible for the methods provided in the related art to obtain the depth information of the stereoscopic unit based solely on the image, that is, it is impossible to determine the degree of protrusion of the stereoscopic unit in the direction perpendicular to the outer surface of the photographed object based solely on the image.

[0057] In order to determine the three-dimensional information of the stereo unit, it is necessary to obtain the camera intrinsic parameters in the related art, and predict the three-dimensional information of the stereo unit through the camera intrinsic parameters and the image. Since the camera intrinsic parameters are internal parameters of the camera, they are affected by factors such as the shooting elements and assembly errors in the camera, and there are individual differences in the camera intrinsic parameters. The camera intrinsic parameters of the camera are difficult to unify. Therefore, in the method of determining the three-dimensional information of the stereo unit through the camera intrinsic parameters and the image, it is necessary to obtain the camera intrinsic parameters corresponding to the camera used to shoot the image. Since in some scenarios, it is difficult or impossible to obtain the camera intrinsic parameters corresponding to the image (such as the camera intrinsic parameters corresponding to the image downloaded from the Internet are difficult to determine), the method of determining the three-dimensional information of the stereo unit in the related art has poor adaptability to different scenarios.

[0058] FIG5 is a schematic diagram of the inventive concept of the three-dimensional information determination method provided in this application.

[0059] The method provided in this application is used to determine the three-dimensional information of a stereoscopic unit protruding from the outer surface of a photographed object based on an image. The inner surface of the stereoscopic unit contacts the outer surface of the photographed object, that is, the inner surface of the stereoscopic unit is located within the outer surface of the photographed object. In the method provided in this application, only the image obtained by photographing the photographed object is obtained, without the need for other external information (such as camera intrinsic parameters), and the three-dimensional information of the stereoscopic unit can be determined, and the size, position, and degree of protrusion of the stereoscopic unit can be obtained.

[0060] In the method for determining 3D information provided in this application, the bounding box information of at least one stereo unit is determined from an image. Subsequently, the 3D information of the stereo unit is determined based on the perspective projection of the stereo unit on the outer surface of the photographed object and the bounding box information of the stereo unit. This method reduces the reliance on external reference information during the 3D information determination process and reduces the difficulty of collecting reference information for determining the 3D information of the stereo unit.

[0061] FIG6 is a schematic diagram showing the principle of determining three-dimensional information provided by an exemplary embodiment of the present application.

[0062] As shown in FIG6 , let h represent the distance from the stereo unit to the camera (i.e., the object distance); d represent the true thickness of the stereo unit (the degree of protrusion of the stereo unit relative to the outer surface of the photographed object), and in FIG6 , |P5P7|=d; w is the true width of the object, and in FIG6 , |P7P8|=w; the projection of the outer surface of the stereo unit on the imaging plane is represented by x out , |P″5P″6|=x out ; The projection of the inner surface of the stereo unit on the imaging plane is x in , P5 and P6 represent the vertices on the outer surface of the stereo unit, P″5 and P″6 represent the vertices of the outer surface of the stereo unit in the imaging plane, and O represents the center point of the camera lens.

[0063] Since light travels in a straight line, when the camera captures the stereo unit, ΔOP″5P″6 and ΔOP5P6 are similar triangles, and ΔOP″6O1 and ΔOP6O2 are also similar triangles. There is a proportional relationship between OP″6 and OP6 (1):

[0064] Where f represents the focal length of the camera. For other parameters, please refer to the above description and will not be repeated here.

[0065] For each stereo unit in the same image, f and h are constants. OP″6 and OP6 are positively correlated with the stereo unit's true thickness d. That is, the greater the true thickness d, the closer the distance between the stereo unit's outer surface and the camera, and the larger the stereo unit's image in the imaging plane. The smaller the true thickness d, the farther the distance between the stereo unit's outer surface and the camera, and the smaller the stereo unit's image in the imaging plane.

[0066] When the actual thickness of the stereo unit is d=0, the inner surface of the stereo unit coincides with the outer surface. Based on the proportional relationship (1) between OP″6 and OP6, the proportional relationship (2) of the inner surface on the imaging plane is obtained:

[0067] Please refer to the above introduction for the meaning of each parameter of this formula, which will not be elaborated here.

[0068] Let (2) / (1) to get the relative convexity between the inner and outer surfaces

[0069] Here, α represents the ratio of the inner and outer surfaces of the stereoscopic elements on the imaging plane (corresponding to the surface ratio in the claims). That is, the ratio α between the inner and outer surfaces on the imaging plane is only related to the object's actual convexity d, and d ∝ (1-α). The relative convexity between stereoscopic elements 1 and 2 on the outer surface of the photographed object is calculated using the following formula.

[0070] The above derivation demonstrates the following: 1. By calculating the ratio of the inner and outer surfaces of a stereo unit, we can determine its degree of convexity. 2. Within the same depth plane (e.g., the outer surface of the photographed object), the relative size and position of multiple stereo units in the image remain consistent with the relative size and position of the units in three-dimensional space. 3. The imaging principle of larger objects appearing closer and smaller objects appearing farther away also applies to stereo units at different depth planes.

[0071] The above conclusions support the prediction of the size and position of the inner and outer surfaces of a stereo unit based on the bounding box information of the stereo unit after obtaining it. Furthermore, the relative protrusion between stereo units at the same depth can be calculated without determining camera intrinsic parameters. The following describes the method for determining three-dimensional information provided by this application through several examples.

[0072] Figure 7 is a flowchart of a method for determining three-dimensional information provided by an exemplary embodiment of the present application. For example, the method may be executed by the computer device 10 in Figure 1 . The following description of the method for determining three-dimensional information will use the computer device as the execution entity. As shown in Figure 2 , the method may include the following steps ( 710 - 730 ).

[0073] Step 710 , obtaining an image obtained by photographing the outer surface of the photographed object; wherein the outer surface of the photographed object has n protruding three-dimensional units, where the three-dimensional unit refers to a solid unit with a three-dimensional structure, and n is a positive integer.

[0074] In some embodiments, the photographed object refers to an object having at least one raised unit on its outer surface. The types of the photographed object include, but are not limited to, at least one of the following: a building, the ground, a plate, etc. Optionally, the outer surface of the photographed object refers to a surface of the photographed object having raised three-dimensional units. For example, if the photographed object is a building, the outer surface of the photographed object refers to the facade of the building. For a building with a three-dimensional rectangular shape, the facade of the building refers to the side surface of the building that is perpendicular to the ground. For another example, if the photographed object is a venue, the outer surface of the photographed object is the ground in the venue. Exemplarily, the image includes a portion of the outer surface of the photographed image, or the image includes the complete outer surface of the photographed image.

[0075] In some embodiments, a stereo unit is a three-dimensional solid having a fixed relative position to the subject being photographed. Optionally, the three-dimensional structure of the stereo unit includes a first surface and a second surface that are parallel to each other. Types of three-dimensional structures of the stereo unit include, but are not limited to, a cuboid, a cylinder, and the like.

[0076] For example, if the subject is a building, the three-dimensional elements include, but are not limited to, windows, balconies, arches, and protruding decorative objects on the building's exterior surface. For example, if the subject is the ground, the three-dimensional elements include, but are not limited to, three-dimensional objects (such as vehicles) and buildings placed on the site. For example, if the subject is a plate, the three-dimensional elements include, but are not limited to, protruding decorative objects and adhesives on the plate's exterior surface.

[0077] Optionally, the stereo unit is fixed to the outer surface of the object being photographed. Exemplarily, the inner surface of the stereo unit is located within the outer surface of the object being photographed. In other words, the inner surface of the stereo unit overlaps with the outer surface of the object being photographed.

[0078] FIG8 is a schematic diagram of a surface of a three-dimensional unit provided by an exemplary embodiment of the present application.

[0079] A stereo unit 810 protrudes from the outer surface 800 of the object being photographed. An outer surface 812 of the stereo unit 810 protrudes from the outer surface 800 of the object being photographed and is parallel to the outer surface 800 of the object being photographed. An outer surface 814 of the stereo unit 810 is on the surface 800 of the object being photographed.

[0080] In some embodiments, a computer device captures an image of the exterior surface of an object captured by a camera, and the image includes at least one complete stereo unit. The camera may include, but is not limited to, a vehicle-mounted camera, a still camera, or the like. Optionally, the image includes images of multiple stereo units, so that the image can provide additional auxiliary information for determining the three-dimensional information of the stereo unit, thereby improving the accuracy of the determined position and size of the stereo unit.

[0081] In some embodiments, the n stereo units include at least one type of stereo units; wherein, stereo units belonging to the same type have the same three-dimensional structure, and stereo units belonging to different types have differences in three-dimensional structure. Optionally, the image obtained by photographing the outer surface of the photographed object includes at least two different types of stereo units, and the degree of protrusion of different stereo units in the direction perpendicular to the outer surface of the photographed object can be the same or different. Optionally, the image is an image taken by a monocular camera. The method provided in the present application has low requirements for the camera, and the image taken by a conventional camera can be used as the image used in the process of determining three-dimensional information in the present application. The image can also be a point cloud image obtained by scanning, so as to carry more image information and improve the accuracy of the determined three-dimensional information.

[0082] In some embodiments, the source of the image includes but is not limited to at least one of the following: photographing the subject with a camera, downloading from the Internet, or uploading by a user.

[0083] For example, in the process of creating a navigation map, the buildings on both sides of the road are photographed by the on-board camera of the vehicle terminal to obtain a street view image; the computer device receives the image sent by the vehicle terminal. For another example, the computer device obtains an overhead image of the road through an aerial photography device. For another example, in the process of creating a virtual scene such as a game, a virtual building needs to be set up. The computer device obtains an image through the Internet and obtains three-dimensional information of the stereo unit included in the image based on the image, which is used to reconstruct the stereo unit in the game scene to obtain a virtual building. Optionally, image attributes such as image resolution are set according to actual needs, and this application does not limit them here.

[0084] Step 720: Process the image to obtain frame information of at least one stereo unit; wherein the frame information is used to locate the first surface and the second surface of the stereo unit in the image, the first surface is parallel to the second surface, and the first surface or the second surface is within the outer surface.

[0085] In some embodiments, the bounding box information of a stereo unit is used to represent the imaging area of ​​the stereo unit in an image. That is, the bounding box information of a stereo unit is used to indicate the imaging area of ​​the stereo unit in an image. As can be seen from the above, a stereo unit has a three-dimensional structure, and therefore the bounding box information of a stereo unit can describe the stereo bounding box of the stereo unit in an image.

[0086] In some embodiments, the first surface and the second surface are the same size, and the distance between the outer surface and the camera is greater than or equal to the distance between the inner surface and the camera. As can be seen from the perspective imaging principle described above, due to perspective distortion, the first surface and the second surface appear different in shape and size in the image.

[0087] Optionally, the frame information of the stereo unit can represent the position and size of the first surface and the second surface of the stereo unit in the image. For example, the frame information of the stereo unit includes the coordinates of at least one positioning point of the first surface in the image, or the coordinates of at least one positioning point of the second surface in the image. For another example, the frame information of the stereo unit includes parameters for calculating the coordinates of at least one positioning point of the first surface in the image, or includes parameters for calculating the coordinates of at least one positioning point of the first surface in the image.

[0088] Exemplarily, the positioning point may be a vertex of the first surface or the second surface, or may refer to a center point of the first surface or the second surface, which is not limited in this application.

[0089] Optionally, one of the first surface and the second surface is an outer surface of the three-dimensional unit, and the other of the first surface and the second surface is an inner surface of the three-dimensional unit. For example, the first surface is an outer surface of the three-dimensional unit, and the second surface is an inner surface of the three-dimensional unit. For another example, the first surface is an inner surface of the three-dimensional unit, and the second surface is an outer surface of the three-dimensional unit.

[0090] In one example, a computer device obtains the border information of a stereo unit based on an image through an artificial intelligence model. For details about this part, please refer to the following embodiments.

[0091] Optionally, in step 720, the computer device determines the frame information corresponding to at least one stereo unit included in the image. For example, the computer device determines the frame information of one stereo unit in the image. For example, the computer device determines the frame information of all n stereo units in the image. For another example, the computer device determines the frame information of multiple stereo units in the image, and the multiple stereo units are located adjacent to each other in the image (the multiple stereo units may be all or part of the n stereo units). Exemplarily, because the position of at least one stereo unit on the outer surface of the photographed object is different, the frame information of the at least one stereo unit is different.

[0092] Step 730 : determining the three-dimensional information of the stereoscopic unit according to the frame information of the stereoscopic unit; wherein the three-dimensional information is used to represent the position of the stereoscopic unit in the image and the degree of protrusion of the stereoscopic unit in a direction perpendicular to the outer surface.

[0093] In some embodiments, the three-dimensional information of the stereo unit is used to describe the position and shape of each surface of the stereo unit. In this step, the computer device only needs the bounding box information of the stereo unit to determine the three-dimensional information of the stereo unit. This method helps to reduce the difficulty of calculating the three-dimensional information of the stereo unit.

[0094] Optionally, the degree of protrusion of a three-dimensional unit in a direction perpendicular to the outer surface refers to the degree of direct protrusion of the three-dimensional unit relative to the outer surface in the direction perpendicular to the outer surface. Optionally, the degree of protrusion of a three-dimensional unit in a direction perpendicular to the outer surface refers to the relative protrusion of a certain three-dimensional unit relative to another three-dimensional unit.

[0095] Optionally, for any one of the at least one stereo unit, the computer device can obtain the three-dimensional information of the stereo unit based on the bounding box information of the stereo unit. Optionally, for any one of the at least one stereo unit, the computer device can jointly determine the three-dimensional information of the stereo unit based on the bounding box information of the at least one stereo unit. For details on the process of determining the three-dimensional information, please refer to the following embodiments.

[0096] In summary, the method provided herein only requires the use of an image of the subject to determine the three-dimensional information of a stereoscopic unit. By processing this image, bounding box information is obtained for locating the position of the stereoscopic unit within the image. Subsequently, based on the imaging patterns of multiple stereoscopic units on the outer surface of the subject, the three-dimensional information of the stereoscopic unit can be obtained solely from the bounding box information of the stereoscopic unit.

[0097] Compared to related technologies that require the use of external reference information such as camera intrinsics to determine the depth information of a stereo unit, this method does not require the acquisition of other external reference information (such as camera intrinsics) when determining the three-dimensional information of a stereo unit. This reduces the amount of reference information required to determine the three-dimensional information of a stereo unit, enabling the determination of the three-dimensional information of a stereo unit solely based on the image. This method helps improve the adaptability of three-dimensional information determination methods to different scenarios and also helps reduce the difficulty of obtaining the three-dimensional information of a stereo unit.

[0098] The following describes a method for determining three-dimensional information of a stereo unit through several embodiments.

[0099] In some embodiments, the bounding box information includes a surface ratio, where the surface ratio is used to represent a size ratio between the first surface and the second surface in the image.

[0100] In some embodiments, the surface ratio is used to characterize the ratio between the projection of the first surface in the image and the projection of the second surface in the image. Optionally, when the first surface and the second surface are both rectangular, the surface ratio can characterize the length ratio between any side of the first surface and the corresponding side of the second surface. As can be seen from the above derivation process, the two surfaces with different depths from the imaging plane (that is, the image mentioned in the specification) conform to the principle of perspective imaging. In other words, the length ratio between any side of the first surface and the corresponding side of the second surface is the same, which is the surface ratio.

[0101] Exemplarily, the computer device obtains the surface ratio from the frame information according to the surface ratio identifier; the surface ratio identifier is used to represent the storage position of the surface ratio in the frame information.

[0102] Step 730 , determining the three-dimensional information of the stereo unit according to the frame information of the stereo unit, may include the following sub-steps (not shown in the drawings of the specification).

[0103] In sub-step 732 , the computer device determines position information of the stereo unit according to the surface ratio of the stereo unit. The position information is used to represent the position of the stereo unit in the image.

[0104] Optionally, the position information of the stereo unit includes at least one of the following: a position of a first surface of the stereo unit in the image, a position of the first surface of the stereo unit in the image.

[0105] As can be seen from the above content, the border information of the stereo unit is used to locate the stereo unit in the image. For example, the border information is used to locate the first surface of the stereo unit and the second surface of the stereo unit in the image. Affected by the shooting environment, the clarity of the first surface of the stereo unit in the image may be different from the clarity of the second surface of the stereo unit in the image. For example, when shooting outdoors in good light, the imaging of the outer surface of the stereo unit in the image is clearer than the imaging of the inner surface of the stereo unit in the image. For another example, when shooting outdoors in good light, if there is an auxiliary light source inside the object being photographed, that is, the light intensity on the inner surface side of the stereo unit is greater than that on the outer surface side of the stereo unit.

[0106] This makes it difficult for a computer to accurately locate both the first and second surfaces of the stereo unit within the image using the bounding box information of the stereo unit determined from the image. In other words, the accuracy of the bounding box information in locating the first surface of the stereo unit within the image is inconsistent with the accuracy of the bounding box information in locating the second surface of the stereo unit within the image.

[0107] In some embodiments, sub-step 732 is used to correct the frame information of the stereo unit according to the surface ratio to obtain position information. Compared with the frame information, the position information can more accurately locate the position of the stereo unit in the image.

[0108] Optionally, the computer device determines the second surface position based on the surface ratio and the first surface position in the bounding box information; wherein the first surface position is used to locate a target surface in the image. The target surface is the surface of the first and second surfaces of the stereo unit for which the bounding box information has a higher positioning accuracy; the computer device obtains the position information of the stereo unit based on the first and second surface positions.

[0109] Exemplarily, the target surface is pre-set, for example, from the first surface and the second surface of the stereo unit, a surface representing the outer surface of the stereo unit is determined as the target surface.

[0110] For details on determining the location information in this step, please refer to the following description.

[0111] In sub-step 734 , the computer device determines convexity information of the three-dimensional unit according to the surface ratio of the three-dimensional unit. The convexity information is used to represent the degree of convexity of the three-dimensional unit in a direction perpendicular to the outer surface.

[0112] The convexity information can be understood as representing the depth information of the stereo unit in the image. Specifically, the convexity information represents the degree of convexity of the outer surface of the stereo unit relative to the outer surface. Optionally, the degree of convexity represented by the convexity information can be a relative degree of convexity or an absolute degree of convexity. The relative degree of convexity represents the degree of protrusion of the stereo unit from the outer surface, while the absolute degree of convexity represents the distance between the outer surface of the stereo unit and the outer surface, that is, the thickness of the stereo unit on the outer screen surface. The absolute degree of convexity can be represented by the absolute degree of convexity. The calculation method of the absolute degree of convexity is described in the following examples.

[0113] Optionally, the surface ratio of a solid unit is negatively correlated with the degree of protrusion of the solid unit in a direction perpendicular to the outer surface. The larger the surface ratio of a solid unit, the smaller the degree of protrusion of the solid unit in a direction perpendicular to the outer surface; and the smaller the surface ratio of a solid unit, the greater the degree of protrusion of the solid unit in a direction perpendicular to the outer surface.

[0114] In sub-step 736 , the computer device obtains three-dimensional information of the stereo unit based on the position information of the stereo unit and the protrusion information of the stereo unit.

[0115] Optionally, the computer device obtains three-dimensional information of the stereo unit based on the position information of the stereo unit and the protrusion information of the stereo unit, including: the computer device combines the position information of the stereo unit and the protrusion information of the stereo unit to obtain the three-dimensional information of the stereo unit. For example, the three-dimensional information of the stereo unit is composed of the position information of the stereo unit and the protrusion information of the stereo unit.

[0116] The above method allows the convexity information of a stereo unit to be calculated using the surface ratio. In other words, all the information needed to determine the convexity information can be obtained from the image, reducing the reliance on external reference information (such as camera intrinsic parameters) in determining the 3D information of the stereo unit and simplifying the conditions for determining the 3D information of the stereo unit.

[0117] The following describes the method for determining the protrusion information through several embodiments.

[0118] In some embodiments, the image includes multiple stereo units; sub-step 734, the computer device determines the protrusion information of the stereo unit according to the surface ratio of the stereo unit, which may include the following steps (not shown in the drawings of the specification).

[0119] Sub-step 734-a, for a first stereo unit among the multiple stereo units included in the image, the computer device determines the relative convexity of the first stereo unit based on the surface ratio of the first stereo unit and the surface ratio of the second stereo unit; wherein the convexity of the second stereo unit in the direction perpendicular to the outer surface is a known reference value, and the relative convexity of the first stereo unit is used to characterize the convexity of the first stereo unit relative to the second stereo unit in the direction perpendicular to the outer surface.

[0120] In some embodiments, the first stereo unit is a stereo unit with frame information among multiple stereo units. As can be seen from the above content, n stereo units are raised on the outer surface of the photographed object, and the image obtained by photographing the outer surface of the photographed object may include m stereo units (corresponding to the multiple stereo units mentioned in sub-step 734-a), where m is a positive integer less than or equal to n. In step 720, the computer device can calculate the frame information of k stereo units among the m stereo units, where k is a positive integer less than or equal to m. That is, the first stereo unit can be any one of the k stereo units.

[0121] To reduce the computational complexity and time required to determine the 3D information of a stereo unit, the computer device selects k stereo units from the m stereo units in the image and calculates 3D information for each of the at least one stereo unit. Specifically, k is typically less than m. For details on methods for selecting at least one stereo unit from an image and determining bounding box information for the at least one stereo unit, please refer to the following embodiments.

[0122] Exemplarily, the computer device uses each of the multiple stereo units as a first stereo unit and executes sub-step 734 .

[0123] In some embodiments, the second three-dimensional unit serves as a reference unit among the plurality of three-dimensional units for calculating the relative protrusion degree. That is, the protrusion degree of the second three-dimensional unit in a direction perpendicular to the outer surface is a known reference value, which may be pre-set. For example, a staff member sets the known reference value based on experience.

[0124] Optionally, the second stereo unit may be any one of the n stereo units, and the image includes the imaging of the second stereo unit. Exemplarily, the computer device determines the border information of at least two stereo units in step 720, and the at least two stereo units include the second stereo unit.

[0125] Exemplarily, the computer device obtains the surface ratio of the first stereo unit from the border information in the first stereo unit, obtains the surface ratio of the second stereo unit from the border information in the second stereo unit, and determines the relative protrusion of the first stereo unit based on the surface ratio of the first stereo unit and the surface ratio of the second stereo unit.

[0126] In some embodiments, the relative convexity of a first three-dimensional unit refers to the degree of protrusion of the first three-dimensional unit relative to the second three-dimensional unit in a direction perpendicular to the outer surface. When both the inner surface of the first three-dimensional unit and the inner surface of the second three-dimensional unit are in contact with the outer surface, the relative convexity of the first three-dimensional unit is the ratio between the actual thickness of the first three-dimensional unit and the actual thickness of the second three-dimensional unit. As can be seen from the derivation shown in Figure 6 above, the relative protrusion of the first three-dimensional unit is related to the surface ratio of the first three-dimensional unit and the surface ratio of the second three-dimensional unit.

[0127] In some embodiments, the relative convexity of the first three-dimensional unit is negatively correlated with the surface ratio of the first three-dimensional unit; and the relative convexity of the first three-dimensional unit is positively correlated with the surface ratio of the second three-dimensional unit.

[0128] Optionally, the larger the surface ratio of the first three-dimensional unit, the smaller the relative convexity of the first three-dimensional unit; the smaller the surface ratio of the first three-dimensional unit, the larger the relative convexity of the first three-dimensional unit. Optionally, the larger the surface ratio of the second three-dimensional unit, the larger the relative convexity of the first three-dimensional unit; the smaller the surface ratio of the second three-dimensional unit, the smaller the relative convexity of the first three-dimensional unit.

[0129] Optionally, the relative protrusion of the first three-dimensional unit is It can be calculated using formula (3) above.

[0130] In sub-step 734 - a , d2 represents the degree of protrusion of the second three-dimensional unit in a direction perpendicular to the outer surface, which is a known reference value; α1 represents the surface ratio of the first three-dimensional unit, and α2 represents the surface ratio of the second three-dimensional unit.

[0131] For example, if d2 is equal to 1.0, α1 is equal to 0.4, and α2 is equal to 0.5, the relative convexity of the first three-dimensional unit is 1.2.

[0132] In sub-step 734 - b , the computer device determines the convexity information of the first 3D unit according to the relative convexity of the first 3D unit.

[0133] Since the degree of protrusion of the second three-dimensional unit in a direction perpendicular to the outer surface is known, the degree of protrusion of the first three-dimensional unit in a direction perpendicular to the outer surface can also be determined by determining the relative protrusion between the first and second three-dimensional units. The relative protrusion of the first three-dimensional unit can be used to represent the degree of protrusion of the first three-dimensional unit in a direction perpendicular to the outer surface.

[0134] Optionally, the computer device determines the relative convexity of the first stereo unit as the convexity information of the first stereo unit, or the computer device determines the product of the relative convexity and a known reference value as the absolute convexity, and uses the absolute convexity as the convexity information of the first stereo unit.

[0135] In some embodiments, the method for determining three-dimensional information further includes: determining category information of at least one stereo unit based on the image, and determining a second stereo unit from the at least one stereo unit based on the category information; wherein the category information is used to characterize the category to which the stereo unit belongs, and stereo units of different categories have different three-dimensional structures. Optionally, the computer device determines a stereo unit whose category information matches a preset category as the second stereo unit. The computer device may determine one second stereo unit from the at least one stereo unit, or may determine multiple second stereo units from the at least one stereo unit. Exemplarily, the category information matching the preset category means that the category information is the same as the preset category.

[0136] In some embodiments, after determining the relative convexity of the first stereo unit, the computer device sets the relative convexity of the first stereo unit to the relative convexity of similar stereo units; a similar stereo unit refers to a stereo unit of the same type as the first stereo unit in at least one stereo unit. In other words, the similar stereo unit and the first stereo unit have the same three-dimensional structure. Because the inner surfaces of the n stereo units all contact the outer surface of the photographed object, and stereo units of the same type have the same three-dimensional structure, this method helps reduce the computational workload of the computer device.

[0137] Compared with the related technologies that require binocular images or camera internal parameters to determine the depth information of the stereo unit in the image, the relative convexity is used to characterize the degree of convexity of the stereo unit in the direction perpendicular to the outer surface. The information for calculating the relative convexity can be obtained directly from the image, which helps to reduce the amount of external reference information that is relied upon when predicting the degree of convexity of the stereo unit based on the image, making the method for determining three-dimensional information applicable to more scenarios.

[0138] The following describes a method for determining the position information of a stereo unit through several embodiments.

[0139] In some embodiments, sub-step 732 , the computer device determines the position information of the stereo unit according to the surface ratio of the stereo unit, which may include the following steps (not shown in the drawings of the specification).

[0140] In sub-step 732-a, the computer device determines a vanishing point position based on the bounding box information of at least one stereo unit; wherein the vanishing point position refers to the position of the vanishing point in the image, and the vanishing point refers to the perspective imaging intersection point of the stereo unit in the image.

[0141] For the introduction of the vanishing point of an image, please refer to the above embodiment, which will not be repeated here. For the method of determining the position of the vanishing point, please refer to the following introduction.

[0142] Sub-step 732-b: For a first stereo unit among at least one stereo unit, the computer device determines a first surface position and a second surface position of the first stereo unit based on the vanishing point position and the surface ratio of the first stereo unit; wherein the first surface position refers to the position of the first surface of the first stereo unit in the image, and the second surface position refers to the position of the second surface of the first stereo unit in the image.

[0143] In some embodiments, the first surface position is used to locate the first surface of the stereo unit in the image. Exemplarily, when the first surface is rectangular, the first surface position includes positions of four vertices of the first surface in the image.

[0144] For example, when the first surface is rectangular, the first surface position includes the position of the first surface's positioning point in the image and the size information of the first surface. In this case, the computer device determines the positions of the first surface's four vertices in the image based on the position of the first surface's positioning point in the image and the size information of the first surface. The positioning point includes, but is not limited to, any one of the first surface's four vertices and the center point of the first surface (i.e., the intersection of two diagonals).

[0145] For example, when the first surface is circular, the first surface position includes the position of the center of the first surface in the image and the radius of the first surface. It should be noted that the representation of the first surface position is determined according to the actual shape of the first surface and is not limited in this application.

[0146] In some embodiments, the second surface position is used to locate the second surface of the stereo unit in the image. Exemplarily, when the second surface is rectangular, the second surface position includes positions of four vertices of the second surface in the image.

[0147] In some embodiments, a computer device determines a first surface position and a second surface position of the first stereo unit based on a vanishing point position and a surface ratio of the first stereo unit, including: the computer device determines the first surface position based on border information of the first stereo unit; the computer device determines a position ratio between the second surface position and the first surface position based on a geometric relationship between the first surface of the first stereo unit and the second surface of the first stereo unit in the image, and the surface ratio of the first stereo unit; wherein the geometric relationship is determined based on the vanishing point position; and the computer device determines the second surface position based on the position ratio and the first surface position.

[0148] Optionally, the bounding box information of the first stereo unit includes a first surface position, a surface ratio, and a center point offset; wherein the center point offset value is used to represent the relative offset between the inner surface and the outer surface of the first stereo unit. Exemplarily, the first surface position includes the positions of four vertices of the first surface in the image, and the computer device directly obtains the first surface position from the bounding box information of the first stereo unit.

[0149] In some embodiments, the geometric relationship between the first surface of the first stereo unit and the second surface of the first stereo unit in the image is used to locate the second surface of the first stereo unit in the image through a perspective projection relationship.

[0150] Optionally, the computer device connects a first vertex and a vanishing point on a first surface of the first solid unit to obtain a first straight line, and a second vertex on the second surface is located on the first straight line, or the second vertex is located on a reverse extension of the first straight line. A border line exists between the first vertex and the second vertex of the first solid unit.

[0151] In some embodiments, the computer device determines the position ratio between the second surface position and the first surface position based on the geometric relationship between the first surface of the first stereo unit and the second surface of the first stereo unit in the image, and the surface ratio of the first stereo unit, including: for any first vertex on the first surface of the first stereo unit, the computer device determines the position of the first vertex on the image based on the first surface position of the first stereo unit; the computer device determines a first distance between the first vertex and the vanishing point on the image based on the position of the first vertex on the image and the position of the vanishing point; the computer determines a second distance between the second vertex on the image and the vanishing point based on the surface ratio of the first stereo unit and the first distance between the first vertex and the vanishing point; the computer device uses the ratio of the first distance to the second distance as the position ratio, and the computer device determines the position of the second vertex based on the position of the first vertex on the image, the vanishing point position and the position ratio; wherein, the first vertex on the image is the image of the first vertex in the image, and the second vertex on the image is the image of the second vertex in the image.

[0152] Optionally, when the three-dimensional structure of the first three-dimensional unit is a cube, the first vertex refers to any vertex on the first surface of the first three-dimensional unit, and the second vertex is a vertex on the second surface of the first three-dimensional unit connected to the first vertex by a border line.

[0153] FIG9 is a schematic diagram of a second surface position determination process provided by an exemplary embodiment of the present application.

[0154] For the first stereo unit 910 in the image 900, the four vertices P″5, P″6, P″9, P″ 10 The position of the image 900 can be determined by the first surface position, the vanishing point Z vp The vanishing point position is known. According to the geometric relationship between the first surface of the first stereo unit 910 and the second surface of the first stereo unit 910 in the image 900, the four vertices P″7, P″8, P″ 11 , P″ 12 Located on the straight line P″5Z vp 、P″6Z vp 、P″9Z vp , P″ 10 Z vp superior.

[0155] Taking the position of P″7 on the image as an example, due to ΔZ vp P″7P″ 11 and ΔZ vp P″5P″9 are similar triangles, so, Wherein, α is the surface ratio of the first three-dimensional unit 910. P″5P″7=(1-α)Z vp P″5, according to P″5, Z vp The position of P″7 on the image can be determined by the same method. 11 , P″ 12 The second surface position of the first stereo unit 910 can be obtained from the position on the image.

[0156] In FIG9 , a method for determining the position of the second surface of the first stereo unit 910 is introduced by taking the first surface of the first stereo unit 910 as the outer surface of the first stereo unit 910 and the first surface of the first stereo unit 910 as the inner surface of the first stereo unit 910 as an example.

[0157] When the first surface of the first stereo unit 910 is the inner surface of the first stereo unit 910 and the first surface of the first stereo unit 910 is the outer surface of the first stereo unit 910, the position of the second surface of the first stereo unit 910 can also be determined. In this case, the position of the vertex P″7 of the inner surface of the first stereo unit 910 on the image is known, and P″5P″7=(1+α)Z vp P″7, the position of the vertex P″5 of the outer surface of the first stereo unit 910 on the image can be determined.

[0158] In one example, because the distance between the outer surface of the stereoscopic unit and the camera is less than or equal to the distance between the outer surface of the stereoscopic unit and the camera, the outer surface of the stereoscopic unit may obscure the inner surface of the stereoscopic unit. This results in poor accuracy in locating the inner surface of the stereoscopic unit in the image when the computer device determines the bounding box information based on the image. Therefore, the first surface of the first stereoscopic unit may be the outer surface of the first stereoscopic unit.

[0159] In another example, the second surface of the first stereoscopic unit can be the inner surface of the first stereoscopic unit. Of course, in situations where the shooting environment is dim, the first surface is borderless, or the first surface has low light reflectivity, the first surface of the first stereoscopic unit can be the inner surface of the first stereoscopic unit, and the second surface of the first stereoscopic unit can be the outer surface of the first stereoscopic unit. It should be noted that whether the first surface is the inner or outer surface of the first stereoscopic unit is determined based on actual needs and is not limited in this application.

[0160] Through the above method, the first surface position of the first surface of the stereo unit, which is relatively accurately located in the frame information, is maintained unchanged, and the second surface position of the stereo unit is determined based on the first surface position, surface ratio, and vanishing point position based on the perspective projection principle. This helps to improve the accuracy of using the position information in the three-dimensional information to locate the first and second surfaces of the stereo unit in the image.

[0161] In sub-step 732 - c , the computer device obtains position information of the first stereo unit based on the first surface position and the second surface position.

[0162] In some embodiments, the computer device obtains the position information of the first stereo unit based on the first surface position and the second surface position, including: the computer device combines the first surface position and the second surface position to obtain the position information of the first stereo unit. Optionally, the position information of the first stereo unit is composed of the first surface position and the second surface position.

[0163] Because the bounding box information of a stereo unit is predicted through deep learning models and other methods, the accuracy of the stereo unit's bounding box information in locating the stereo unit in the image may be low, affected by factors such as the deep learning model's recognition accuracy and generalization ability. The above method uses the vanishing point combined with the stereo unit's bounding box information to determine the stereo unit's position information. This is equivalent to adjusting and optimizing the less accurate information in the stereo unit's bounding box information, which helps to improve the accuracy of the stereo unit's position information in locating the stereo correspondence in the image, thereby representing the accuracy of the determined three-dimensional information of the stereo unit.

[0164] The following describes a method for calculating the vanishing point position through several embodiments.

[0165] In some embodiments, sub-step 732 - a , the computer device determines the vanishing point position based on the bounding box information of at least one stereo unit, which may include the following steps.

[0166] For each of the at least one stereo unit, the computer device determines, based on bounding box information of the stereo unit, at least one perspective imaging line of the stereo unit in the image; wherein the perspective imaging line is used to represent a perspective imaging direction when photographing the stereo unit; the computer device determines, based on the perspective imaging lines of each stereo unit, at least one candidate point, where the candidate point is an intersection of at least two perspective imaging lines; the computer device determines a vanishing point from the at least one candidate point based on distances between the candidate point and each perspective imaging line; and the computer device determines a position of the vanishing point in the image as a vanishing point position.

[0167] In some embodiments, the computer device determines at least one perspective imaging line of the stereo unit in the image based on the border information of the stereo unit, including: the computer device determines the positions of the first vertex and the second vertex on the image based on the border information of the stereo unit, and the computer device determines the straight line passing through the first vertex and the second vertex in the image as the perspective imaging line.

[0168] The first vertex is located on the first surface of the stereo unit, and the second vertex is located on the second surface of the stereo unit. The first vertex and the second vertex are located at the same position on the first surface of the stereo unit and the second vertex on the first surface of the stereo unit. For example, the first vertex is the vertex at the lower left corner of the first surface of the stereo unit, and the second vertex is the vertex at the lower left corner of the second surface of the stereo unit. The coordinates of the first vertex and the second vertex on the image can be determined using the border information, and the analytical expression of the perspective imaging line is calculated based on the coordinates of the first vertex and the second vertex, thereby determining the perspective imaging line.

[0169] Optionally, for any one of the at least one stereo units, the computer device determines, based on the bounding box information of the stereo unit, p perspective imaging lines of the stereo unit in the image. p is a positive integer. For example, p is equal to 1, 2, 3, or 4. Optionally, for two different stereo units, the number of perspective imaging lines determined by the computer device based on the bounding box information of the two stereo units may be the same or different.

[0170] For example, the number of perspective imaging lines determined by the computer device based on the bounding box information of the second stereo unit is greater than the number of perspective imaging lines determined based on the bounding box information of the non-reference stereo unit; wherein the non-reference stereo unit refers to a stereo unit other than the second stereo unit in the at least one stereo unit. This method helps improve the correlation between the determined vanishing point position and the second stereo unit, and helps improve the accuracy of the relative convexity determined based on the second stereo unit in the above embodiment.

[0171] In some embodiments, the computer device determines at least one candidate point based on the perspective imaging lines of each stereo unit, including: for any two perspective imaging lines among multiple perspective imaging lines, the computer device determines the intersection of the two perspective imaging lines and uses the intersection as a candidate point.

[0172] In some embodiments, a computer device determines a vanishing point from at least one candidate point based on the distances between the candidate point and each perspective imaging line, including: if the number of neighboring lines possessed by the candidate point satisfies a first condition, the computer device determines the candidate point as a vanishing point, where the neighboring line refers to a perspective imaging line whose distance from the candidate point is less than or equal to a first threshold; or, the computer device calculates the sum of the distances between the candidate point and each perspective imaging line; if the sum of the distances between the candidate point and each perspective imaging line is less than or equal to a second threshold, the computer device determines the candidate point as a vanishing point.

[0173] Optionally, the first condition is used to determine the vanishing point based on the number of adjacent lines of the candidate point. The first condition includes but is not limited to at least one of the following: the number of adjacent lines of the candidate point is the largest, and the number of adjacent lines of the candidate point is greater than or equal to a third threshold.

[0174] In one example, assuming that at least one candidate point includes a candidate points, where a is a positive integer, and a computer device determines b perspective imaging lines based on bounding box information of at least one stereo unit, where b is a positive integer greater than or equal to 2. The computer device determines a vanishing point from the a candidate points based on the distances between the candidate point and each perspective imaging line, including: for each of the a candidate points, the computer device determines the distance between the candidate point and any one of the b perspective imaging lines to obtain b distances; the computer device determines the perspective imaging line corresponding to a distance greater than a first threshold among the b distances as a neighboring line of the candidate point, and the computer device counts the number of neighboring lines of the candidate point. Based on the number of neighboring lines for each of the a candidate points, the computer device selects the candidate point with the largest number of neighboring lines as the vanishing point.

[0175] In another example, assuming that at least one candidate point includes c candidate points, where c is a positive integer, and the computer device determines d perspective imaging lines based on bounding box information of at least one stereo unit, where d is a positive integer greater than or equal to 2. The computer device determines a vanishing point from the c candidate points based on the distances between the candidate point and each perspective imaging line, including: the computer device determines the distances between a first candidate point from the c candidate points and each of the d perspective imaging lines, obtaining d distances; the computer device adds the d distances to obtain the sum of the distances between the first candidate point and the d perspective imaging lines; if the sum of the distances between the first candidate point and the d perspective imaging lines is less than or equal to a second threshold, the computer device determines the candidate point as a vanishing point and selects the candidate point with the largest number of neighboring lines as the vanishing point; if the sum of the distances between the first candidate point and the d perspective imaging lines is less than or equal to the second threshold, the computer device removes the first candidate point from the c candidate points and determines a new first candidate point from the remaining candidate points in the c candidate points, repeating the above steps until a vanishing point is determined from the c candidate points.

[0176] It should be noted that the first threshold and the second threshold are both positive numbers. The specific values ​​of the first threshold and the second threshold are set according to actual needs and are not limited in this application.

[0177] FIG. 10 is a schematic diagram of a vanishing point determination process provided by an exemplary embodiment of the present application.

[0178] As shown in Figure 10 , the bounding box information of stereo unit 1010 can be used to locate the outer and inner surfaces of stereo unit 1010 within the image. Based on the bounding box information of stereo unit 1010, the computer device determines a perspective imaging line 1020 and uses the intersection 1030 of perspective imaging line 1020 with other perspective imaging lines as a candidate point. After obtaining at least one candidate point, the computer device determines a vanishing point from the at least one candidate point.

[0179] By determining the vanishing point using the bounding box information of each stereo unit, this method can correct inaccuracies in the bounding box information based on the principle of perspective imaging. This makes the three-dimensional information more capable of locating the stereo unit, helping to improve the accuracy of the determined three-dimensional information of the stereo unit.

[0180] FIG. 11 is a schematic diagram showing a comparison of border information and position information provided by an exemplary embodiment of the present application.

[0181] As shown in Figure 11 , through the description of the position information determination process in the above embodiment, the computer device determines the vanishing point position and, based on the vanishing point position, determines the second surface of the stereo unit (in this embodiment, the second surface of the stereo unit), thereby obtaining the position information of each stereo unit. Specifically, at least one perspective projection line determined based on the stereo unit's position information intersects at the vanishing point VP. Before vanishing point adjustment, the stereo units in image 1110 had multiple vanishing points. After the above adjustment, the vanishing points of the stereo units in image 1120 are unified.

[0182] The following describes a method for determining the border information of a three-dimensional unit through several embodiments.

[0183] In some embodiments, step 720, processing the image to obtain frame information of at least one stereo unit, may include the following sub-steps (not shown in the drawings of the specification).

[0184] In sub-step 722 , the computer device crops the image to obtain a cropped image including at least one stereo unit.

[0185] In some embodiments, since the image includes multiple stereo units, in order to reduce the amount of computation required by the computer device during image processing, the image needs to be cropped so that the number of stereo units included in the cropped image is less than the number of stereo units included in the image, that is, the number of stereo units whose three-dimensional information needs to be determined is reduced.

[0186] In some embodiments, in sub-step 722, the computer device crops the image to obtain a cropped image including at least one stereo unit, including: the computer device obtains cropping area information for the image, the cropping area information including the positions of four cropping vertices of the cropped image in the image; the computer device removes other areas in the image except the cropping area obtained by connecting the four cropping vertices to obtain the cropped image.

[0187] In some embodiments, the clipping region information includes the coordinates of the four clipping vertices of the clipping image in the image. Optionally, the clipping region information is obtained by manually determining the coordinates of the four clipping vertices of the clipping image in the image.

[0188] Optionally, the cropping area information is machine-labeled. Exemplarily, a computer device inputs an image into a cropping area determination model, which identifies stereo units from the image and calculates the cropping area information based on the imaging area of ​​the stereo units in the image. For example, after the cropping area determination model identifies m stereo units from the image, it determines the imaging area of ​​the m stereo units in the image, and determines the minimum area of ​​the imaging area of ​​the m stereo units in the image as the cropping area. It should be noted that the cropping area determination model refers to any model that can perform image recognition, and this application does not limit the type of the cropping area determination model.

[0189] For example, the computer device removes all areas of the image except the clipping area formed by connecting the four clipping vertices, and obtains a clipped image including m stereo units, where m is a positive integer. Due to perspective distortion, the clipped image may be an irregular trapezoid.

[0190] In sub-step 724 , the computer device performs perspective transformation on the cropped image to obtain a corrected image, wherein the shape of the first surface of the stereo unit in the corrected image is the same as the shape of the first surface in the three-dimensional structure of the stereo unit.

[0191] In some embodiments, in sub-step 724, the computer device performs a perspective transformation on the cropped image to obtain a corrected image, including: determining, based on the region information of the cropped image, region information of the corrected image; wherein the region information of the cropped image is used to represent the positions of four vertices of the cropped image in the image, and the region information of the corrected image is used to represent the positions of four vertices of the corrected image in the image; calculating, based on the region information of the cropped image and the region information of the corrected image, a transformation matrix; wherein the transformation matrix is ​​used to transform pixels in the cropped image into pixels in the corrected image; determining, for each pixel in the cropped image, the position of the pixel in the corrected image based on the position of the pixel in the cropped image and the transformation matrix; and arranging each pixel based on the position of the pixel in the corrected image to obtain the corrected image.

[0192] FIG12 is a schematic diagram of a corrected stereo unit provided by an exemplary embodiment of the present application. The corrected image represents perspective imaging when the outer surface of the photographed object is parallel to the imaging plane. As shown in FIG12 , the surface P″1P″2P″4P″3 represents the outer surface of the photographed object, and the outer surface of the photographed object is parallel to the imaging plane. The three-dimensional structure of the stereo unit 1210 is a rectangular parallelepiped. In the corrected image, the upper surface P″5P″6P″7P″8 of the stereo unit 1210 is perpendicular to the outer surface P″1P″2P″4P″3 of the photographed object.

[0193] It should be understood that in the border information of the stereo unit subsequently determined using the corrected image, the border information is used to locate the stereo unit in the corrected image. Since the corrected image is equivalent to an image taken when the perspective imaging plane and the outer surface are viewed, the border information can also locate the actual position of the stereo unit on the outer surface.

[0194] In some embodiments, the corrected image is rectangular. Optionally, the corrected image is shaped like a circumscribed quadrilateral of the cropped image. The computer device determines the maximum horizontal coordinate, the minimum horizontal coordinate, the maximum vertical coordinate, and the minimum horizontal and vertical coordinates from the region information of the cropped image. The computer device combines the maximum horizontal coordinate, the minimum horizontal coordinate, the maximum vertical coordinate, and the minimum horizontal and vertical coordinates based on the corrected image being shaped like a circumscribed quadrilateral of the cropped image to obtain the region information of the corrected image.

[0195] FIG13 is a schematic diagram of image cropping correction provided by an exemplary embodiment of the present application.

[0196] As shown in Figure 13, the computer device crops the image obtained by capturing the outer surface of the object to obtain a cropped image. The computer device then performs perspective correction on the cropped image to obtain a corrected image. This cropping and correction operation preserves the desired image while removing distortion caused by the capture process, facilitating subsequent detection and identification of stereo units from the corrected image.

[0197] In some embodiments, the position of a point in an image can be represented using two-dimensional coordinates, where the abscissa represents the horizontal position of the point in the image, and the ordinate represents the vertical position of the point in the image. During the perspective transformation process, to ensure that the parameters in the transformation matrix are solvable, a new dimension is added to the coordinates of the point in the image, and the coordinate value of any point in the cropped image in this dimension is fixed, for example, equal to 1.

[0198] In one example, the homogeneous coordinates of the four vertices of the clipped image in the image are (u i ,v i ,w)(i=1,2,3,4), where w=1. The homogeneous coordinates of the four vertices of the rectified image are (x i ,y i ,w')(i=1,2,3,4), let w'=1.

[0199] The general transformation formula for perspective transformation:

[0200] Where [u,v,w] is the position of the vertex of the cropped image in the image, [x,y,w'] is the position of the vertex of the corrected image in the image, is the transformation matrix. Substituting the four vertex coordinates of the cropped image and the four vertex coordinates of the rectified image into the general transformation formula, we can obtain 8 equations.

[0201] Solving the eight simultaneous equations above calculates the values ​​of the parameters in the transformation matrix, resulting in the transformation matrix. The computer then uses the transformation matrix and the universal transformation formula to determine the position of each pixel in the cropped image on the corrected image.

[0202] Correcting the image distortion helps reduce the deformation of the stereo unit in the image due to perspective distortion, and helps improve the accuracy of the determined border information of the stereo unit.

[0203] In sub-step 726 , the computer device processes the rectified image to obtain frame information of at least one stereo unit.

[0204] In some embodiments, sub-step 726 , the computer device processes the rectified image to determine the border information of at least one stereo unit, which may include the following sub-steps (not shown in the drawings of the specification).

[0205] In sub-step 726 - a , the computer device performs feature extraction on the rectified image through the feature extraction layer of the border detection model to obtain feature information of the rectified image.

[0206] In some embodiments, the border detection model is a deep learning model that is used to identify the borders of the stereo unit from the image, that is, the border information of the stereo unit. Exemplarily, the border detection model adopts the structure of the Faster RCNN (Faster Regions with CNN features) model.

[0207] For example, the feature extraction layers (Conv layers) include convolutional (conv), activation (ReLU) layers, and pooling layers. The border recognition layer extracts feature information from the rectified image through convolution, activation, and pooling. This feature information, also known as feature maps, is shared by subsequent layers in the border detection model.

[0208] In sub-step 726-b, the computer device processes the feature information through the border recognition layer of the border detection model to determine at least one target area of ​​the corrected image; wherein the target area is used to indicate the area in the corrected image where the stereo unit is displayed.

[0209] The border recognition layer is also called the region proposal network, which is used to detect the target area in the corrected image. Optionally, the border recognition layer processes the feature information based on softmax to determine whether the corrected image shows a stereo unit in multiple preset anchor areas, and then uses bounding box regression to adjust the border of the anchor area to obtain accurate target areas (proposals). Exemplarily, the vertex position of the anchor area in the corrected image is determined, and the anchor area can usually be understood as a rectangular box. Multiple anchor areas refer to multiple rectangular boxes of different sizes.

[0210] In sub-step 726-c, the computer device determines at least one unit feature information based on the at least one target region and the feature information through a pooling layer of the bounding box detection model. The unit feature information is used to represent the feature information of the stereo unit. The pooling layer is also called (Roi Pooling).

[0211] In sub-step 726 - d , the computer device performs linear mapping on each unit feature information through the linear mapping layer of the bounding box detection model to obtain bounding box information of at least one stereo unit.

[0212] Optionally, the frame information of the 3D unit includes the outer surface position (Outer Box) of the 3D unit, the center offset (Center Offset) from the outer surface to the center of the inner surface, and the size ratio (Inner Scale) of the inner surface relative to the outer surface.

[0213] FIG14 is a schematic diagram of a border information determination process provided by an exemplary embodiment of the present application.

[0214] The computer device inputs the corrected building body into a bounding box detection model. The computer device then extracts features from the corrected image using the feature extraction layer of the bounding box detection model to obtain feature information for the corrected image. The computer device then processes the feature information using the bounding box recognition layer of the bounding box detection model to determine at least one target region in the corrected image. The computer device then determines at least one unit feature information based on the at least one target region and the feature information using the pooling layer of the bounding box detection model. The computer device then linearly maps each unit feature information using the linear mapping layer of the bounding box detection model to obtain bounding box information for at least one stereo unit.

[0215] FIG15 is a schematic diagram of a method for determining three-dimensional information provided by an exemplary embodiment of the present application.

[0216] The computer device obtains an image of the outer surface of the photographed object; crops the image to obtain a cropped image including at least one stereo unit; and performs a perspective transformation on the cropped image to obtain a corrected image. The computer device then processes the corrected image using a bounding box detection model to obtain bounding box information for the at least one stereo unit. The computer device determines the vanishing point position based on the bounding box information of the at least one stereo unit, and determines the second surface position of each stereo unit based on the vanishing point position and the bounding box information. Furthermore, the computer device determines the convexity information of the stereo unit based on the surface ratio in the bounding box information. The computer device determines the three-dimensional information of the stereo unit based on the first surface position, the second surface position, and the convexity information.

[0217] The following describes a method for determining three-dimensional information of a building unit, using a building as an example. This example includes the following steps.

[0218] In step A10, the computer device obtains an image obtained by photographing the outer surface of the object.

[0219] In this example, the subject being photographed is a building, and the exterior surface of the subject being photographed is the building's facade. The three-dimensional units are architectural units. For example, architectural units include, but are not limited to, three-dimensional structures protruding from the building's facade, such as bay windows, balconies, and arches.

[0220] In step A20, the computer device crops the image to obtain a cropped image including at least one building unit. Optionally, the cropping area information used for cropping the image can be manually annotated or automatically recognized by a machine.

[0221] In step A30 , the computer device performs perspective transformation on the cropped image to obtain a corrected image, wherein the shape of the first surface of the building unit in the corrected image is the same as the shape of the first surface in the three-dimensional structure of the building unit.

[0222] Step A40: Process the corrected image to obtain frame information of at least one building unit.

[0223] Optionally, the computer device processes the corrected image through a border detection model, which may specifically include the following steps: performing feature extraction on the corrected image through the feature extraction layer of the border detection model to obtain feature information of the corrected image; performing processing based on the feature information through the border recognition layer of the border detection model to determine at least one target area of ​​the corrected image; determining at least one unit feature information based on at least one target area and feature information through the pooling layer of the border detection model; and performing linear mapping on each unit feature information through the linear mapping layer of the border detection model to obtain border information of at least one building unit.

[0224] In step A50, the corrected image includes multiple building units. For a first building unit among the multiple building units included in the image, the computer device determines the relative protrusion of the first building unit based on the surface ratio of the first building unit and the surface ratio of the second building unit. The computer device determines the protrusion information of the first building unit based on the relative protrusion of the first building unit.

[0225] In step A60, the computer device determines the location of the vanishing point based on the frame information of at least one building unit.

[0226] Step A70: For a first building unit in the at least one building unit, the computer device determines a first surface position and a second surface position of the first building unit based on the vanishing point position and the surface ratio of the first building unit. The computer device obtains position information of the first building unit based on the first surface position and the second surface position.

[0227] In step A80, the computer device obtains three-dimensional information of the building unit based on the position information of the building unit and the protrusion information of the building unit.

[0228] For parts not described in detail in this embodiment, please refer to the above embodiments and will not be described in detail here.

[0229] It should be noted that the method for determining three-dimensional information provided in this application can also be used to determine three-dimensional information for other types of stereoscopic units. For example, in an exemplary application scenario: the subject being photographed is the ground, the stereoscopic units are buildings within the site, the image is a bird's-eye view of the ground, and the degree of protrusion represents the building's height above the ground. The three-dimensional information of the buildings determined in this scenario can be used to construct a three-dimensional environment for an electronic map.

[0230] In another exemplary application scenario, the subject is a highway, the image is an aerial ground image, the stereo units are vehicles on the highway, and the degree of protrusion represents the vehicle's height. The 3D information of the vehicles determined in this scenario can be used to construct a 3D environment in assisted or autonomous driving scenarios. The specific steps for determining the 3D information of the stereo units in these scenarios are similar to the method for determining the 3D information of the building units in this example and are not detailed here.

[0231] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0232] Figure 16 shows a block diagram of a device for determining 3D information captured in an exemplary embodiment of the present application. This device can be implemented as all or part of a 3D information determination device through software, hardware, or a combination of both. Device 1600 may include an image acquisition module 1610, a bounding box determination module 1620, and a 3D determination module 1630.

[0233] The image acquisition module 1610 is used to acquire an image obtained by photographing the outer surface of the photographed object; wherein the outer surface of the photographed object has n protruding three-dimensional units, and the three-dimensional unit refers to a solid unit with a three-dimensional structure, and n is a positive integer.

[0234] The border determination module 1620 is used to process the image to obtain border information of at least one of the three-dimensional units; wherein the border information is used to locate the first surface and the second surface of the three-dimensional unit in the image, the first surface is parallel to the second surface, and the first surface or the second surface is within the outer surface.

[0235] The three-dimensional determination module 1630 is used to determine the three-dimensional information of the three-dimensional unit based on the frame information of the three-dimensional unit; wherein the three-dimensional information is used to represent the position of the three-dimensional unit in the image and the degree of protrusion of the three-dimensional unit in the direction perpendicular to the outer surface.

[0236] In some embodiments, the border information includes a surface ratio, and the surface ratio is used to characterize the size ratio of the first surface and the second surface in the image; the three-dimensional determination module 1630 includes: a position determination submodule, used to determine the position information of the three-dimensional unit according to the surface ratio of the three-dimensional unit, and the position information is used to characterize the position of the three-dimensional unit in the image; a convexity determination submodule, used to determine the convexity information of the three-dimensional unit according to the surface ratio of the three-dimensional unit, and the convexity information is used to characterize the degree of convexity of the three-dimensional unit in the direction perpendicular to the outer surface; a three-dimensional determination submodule, used to obtain the three-dimensional information of the three-dimensional unit based on the position information of the three-dimensional unit and the convexity information of the three-dimensional unit.

[0237] In some embodiments, the image includes a plurality of the stereoscopic units, including a first stereoscopic unit and a second stereoscopic unit, wherein the degree of protrusion of the second stereoscopic unit in a direction perpendicular to the outer surface is a known reference value; the protrusion determination submodule is configured to determine the relative protrusion of the first stereoscopic unit based on the surface proportion of the first stereoscopic unit and the surface proportion of the second stereoscopic unit; wherein the relative protrusion of the first stereoscopic unit is used to characterize the degree of protrusion of the first stereoscopic unit relative to the second stereoscopic unit in a direction perpendicular to the outer surface; and the protrusion information of the first stereoscopic unit is determined based on the relative protrusion of the first stereoscopic unit.

[0238] In some embodiments, the relative convexity of the first three-dimensional unit is negatively correlated with the surface ratio of the first three-dimensional unit; the relative convexity of the first three-dimensional unit is positively correlated with the surface ratio of the second three-dimensional unit.

[0239] In some embodiments, the position determination submodule includes: a vanishing point determination unit, configured to determine a vanishing point position based on border information of at least one of the stereo units; wherein the vanishing point position refers to the position of the vanishing point in the image, and the vanishing point refers to the perspective imaging intersection of the stereo unit in the image; a position calculation unit, configured to determine, for a first stereo unit in at least one of the stereo units, a first surface position and a second surface position of the first stereo unit based on the vanishing point position and the surface ratio of the first stereo unit; wherein the first surface position refers to the position of the first surface of the first stereo unit in the image, and the second surface position refers to the position of the second surface of the first stereo unit in the image; and a position determination unit, configured to obtain position information of the first stereo unit based on the first surface position and the second surface position.

[0240] In some embodiments, the position calculation unit is configured to determine the first surface position based on the border information of the first stereo unit; determine a position ratio between the second surface position and the first surface position based on a geometric relationship between the first surface of the first stereo unit and the second surface of the first stereo unit in the image, and a surface ratio of the first stereo unit; wherein the geometric relationship is determined based on the vanishing point position; and determine the second surface position based on the position ratio and the first surface position.

[0241] In some embodiments, the vanishing point determination unit includes a line determination subunit for determining, for each of at least one of the stereo units, at least one perspective imaging line of the stereo unit in the image based on bounding box information of the stereo unit; wherein the perspective imaging line is used to represent a perspective imaging direction when photographing the stereo unit; a candidate point determination subunit for determining, based on the perspective imaging lines of each of the stereo units, at least one candidate point, where the candidate point is an intersection of at least two perspective imaging lines; a vanishing point determination subunit for determining the vanishing point from the at least one candidate point based on a distance between the candidate point and each of the perspective imaging lines; and a position determination subunit for determining the position of the vanishing point in the image as the vanishing point position.

[0242] In some embodiments, the vanishing point determination subunit is configured to determine the candidate point as the vanishing point if the number of neighboring lines of the candidate point satisfies a first condition, where the neighboring line refers to a perspective imaging line whose distance from the candidate point is less than or equal to a first threshold; or to calculate the sum of the distances between the candidate point and each of the perspective imaging lines; and determine the candidate point as the vanishing point if the sum of the distances between the candidate point and each of the perspective imaging lines is less than or equal to a second threshold.

[0243] In some embodiments, the border determination module 1620 includes: a cropping submodule for cropping the image to obtain a cropped image including at least one of the stereoscopic units; a transformation submodule for performing perspective transformation on the cropped image to obtain a corrected image, wherein the shape of the first surface of the stereoscopic unit in the corrected image is the same as the shape of the first surface in the three-dimensional structure of the stereoscopic unit; and a border determination submodule for processing the corrected image to obtain border information of at least one of the stereoscopic units.

[0244] In some embodiments, the border determination submodule is used to: extract features from the corrected image through the feature extraction layer of the border detection model to obtain feature information of the corrected image; determine at least one target area of ​​the corrected image based on the feature information through the border recognition layer of the border detection model; wherein the target area is used to indicate the area in the corrected image where the stereo unit is displayed; determine at least one unit feature information based on the at least one target area and the feature information through the pooling layer of the border detection model; wherein the unit feature information is used to characterize the feature information of the stereo unit; and linearly map each unit feature information through the linear mapping layer of the border detection model to obtain border information of at least one stereo unit.

[0245] In some embodiments, the transformation submodule is used to determine the regional information of the corrected image based on the regional information of the cropped image; wherein the regional information of the cropped image is used to represent the positions of the four vertices of the cropped image in the image, and the regional information of the corrected image is used to represent the positions of the four vertices of the corrected image in the image; based on the regional information of the cropped image and the regional information of the corrected image, a transformation matrix is ​​calculated; wherein the transformation matrix is ​​used to transform the pixel points in the cropped image into the pixel points in the corrected image; for each pixel point in the cropped image, the position of the pixel point in the corrected image is determined based on the position of the pixel point in the cropped image and the transformation matrix; and based on the position of each pixel point in the corrected image, each pixel point is arranged to obtain the corrected image.

[0246] In some embodiments, the cropping submodule is used to obtain cropping area information for the image, and the cropping area information includes the positions of four cropping vertices of the cropped image in the image; and removes other areas of the image except the cropping area obtained by connecting the four cropping vertices to obtain the cropped image.

[0247] It should be noted that the device provided in the above embodiment, when implementing its functions, only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here. For the beneficial effects of the device provided in the above embodiment, please refer to the description of the method side embodiment, which will not be repeated here.

[0248] Figure 17 shows a block diagram of a computer device provided by an exemplary embodiment of the present application. The three-dimensional information determination device 1700 may be the computer device described above.

[0249] Typically, the computer device 1700 includes a processor 1701 and a memory 1702 .

[0250] The processor 1701 may include one or more processing cores, such as a 4-core processor, a 17-core processor, and the like. The processor 1701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0251] Memory 1702 may include one or more computer-readable storage media, which may be tangible and non-transitory. Memory 1702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1702 stores at least one program, which is loaded and executed by processor 1701 to implement the three-dimensional information determination method provided in each of the above method embodiments.

[0252] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. The computer program is loaded and executed by a processor to implement the three-dimensional information determination method provided by the above-mentioned method embodiments.

[0253] The computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above-mentioned ones.

[0254] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor reads and executes the computer program from the computer-readable storage medium to implement the three-dimensional information determination method provided in the above-mentioned method embodiments.

[0255] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0256] It should be noted that this application may display a prompt interface, pop-up window, or output a voice prompt message before and during the process of collecting the user's relevant data. The prompt interface, pop-up window, or voice prompt message is used to remind the user that their relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining the user's relevant data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining the user's relevant data are terminated, that is, the user's relevant data is not obtained. In other words, the images obtained by photographing the outer surface of the photographed object in this application are collected with the user's consent and authorization in accordance with the requirements of relevant national laws and regulations, and the informed consent or separate consent of the personal information subject is obtained. Subsequent data use and processing are carried out within the scope of the laws and regulations and the authorization of the personal information subject, and the collection, use, and processing of relevant user data must comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0257] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switches, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for determining three-dimensional information, the method being executed by a computer device, the method comprising: Acquire an image obtained by photographing the outer surface of the photographed object; wherein the outer surface of the photographed object has n protruding three-dimensional units, the three-dimensional unit refers to a solid unit with a three-dimensional structure, and n is a positive integer; Processing the image to obtain frame information of at least one of the three-dimensional units; wherein the frame information is used to locate a first surface and a second surface of the three-dimensional unit in the image, the first surface is parallel to the second surface, and the first surface or the second surface is within the outer surface; The three-dimensional information of the three-dimensional unit is determined according to the frame information of the three-dimensional unit; wherein the three-dimensional information is used to characterize the position of the three-dimensional unit in the image and the protrusion degree of the three-dimensional unit in a direction perpendicular to the outer surface.

2. The method according to claim 1, wherein the frame information comprises a surface ratio, and the surface ratio is used to characterize a size ratio between the first surface and the second surface in the image; The determining the three-dimensional information of the stereoscopic unit according to the frame information of the stereoscopic unit includes: Determining position information of the stereoscopic unit according to a surface ratio of the stereoscopic unit, wherein the position information is used to represent a position of the stereoscopic unit in the image; Determining, according to the surface ratio of the three-dimensional unit, protrusion information of the three-dimensional unit, wherein the protrusion information is used to characterize the protrusion degree of the three-dimensional unit in a direction perpendicular to the outer surface; Based on the position information of the three-dimensional unit and the protrusion information of the three-dimensional unit, three-dimensional information of the three-dimensional unit is obtained.

3. The method according to claim 2, wherein the image comprises a plurality of the three-dimensional units, wherein the plurality of the three-dimensional units comprises a first three-dimensional unit and a second three-dimensional unit, and a protrusion degree of the second three-dimensional unit in a direction perpendicular to the outer surface is a known reference value; The step of determining the protrusion information of the three-dimensional unit according to the surface ratio of the three-dimensional unit includes: Determining a relative convexity of the first three-dimensional unit according to a surface ratio of the first three-dimensional unit and a surface ratio of the second three-dimensional unit; wherein the relative convexity of the first three-dimensional unit is used to characterize a convexity of the first three-dimensional unit relative to the second three-dimensional unit in a direction perpendicular to the outer surface; The convexity information of the first three-dimensional unit is determined according to the relative convexity of the first three-dimensional unit.

4. The method according to claim 3, The relative protrusion of the first three-dimensional unit is negatively correlated with the surface ratio of the first three-dimensional unit; The relative protrusion of the first three-dimensional unit is positively correlated with the surface ratio of the second three-dimensional unit.

5. The method according to claim 2, wherein determining the position information of the three-dimensional unit according to the surface ratio of the three-dimensional unit comprises: Determine a vanishing point position according to the frame information of at least one of the three-dimensional units; wherein the vanishing point position refers to the position of the vanishing point in the image, and the vanishing point refers to the perspective imaging intersection of the three-dimensional unit in the image; For a first stereo unit among at least one of the stereo units, a first surface position and a second surface position of the first stereo unit are determined according to the vanishing point position and the surface ratio of the first stereo unit; wherein the first surface position refers to a position of a first surface of the first stereo unit in the image, and the second surface position refers to a position of the second surface of the first stereo unit in the image; Based on the first surface position and the second surface position, position information of the first three-dimensional unit is obtained.

6. The method according to claim 5, wherein determining the first surface position and the second surface position of the first three-dimensional unit according to the vanishing point position and the surface ratio of the first three-dimensional unit comprises: Determining the first surface position according to the frame information of the first three-dimensional unit; Determine a position ratio between a position of the second surface of the first stereo unit and a position of the first surface according to a geometric relationship between a first surface of the first stereo unit and a second surface of the first stereo unit in the image and a surface ratio of the first stereo unit; wherein the geometric relationship is determined based on the vanishing point position; The second surface position is determined according to the position ratio and the first surface position.

7. The method according to claim 5, wherein determining the vanishing point position according to the frame information of at least one of the three-dimensional units comprises: For each of the at least one stereoscopic unit, determining at least one perspective imaging line of the stereoscopic unit in the image according to the frame information of the stereoscopic unit; wherein the perspective imaging line is used to represent the perspective imaging direction when photographing the stereoscopic unit; Determine at least one candidate point according to the perspective imaging lines of each of the stereoscopic units, wherein the candidate point is an intersection point of at least two of the perspective imaging lines; Determining the vanishing point from the at least one candidate point according to the distance between the candidate point and each of the perspective imaging lines; The position of the vanishing point in the image is determined as the vanishing point position.

8. The method according to claim 7, wherein determining the vanishing point from the at least one candidate point according to the distance between the candidate point and each of the perspective imaging lines comprises: If the number of neighboring lines of the candidate point satisfies a first condition, the candidate point is determined as the vanishing point, and the neighboring line refers to a perspective imaging line whose distance to the candidate point is less than or equal to a first threshold; or; The sum of the distances between the candidate point and each of the perspective imaging lines is calculated; if the sum of the distances between the candidate point and each of the perspective imaging lines is less than or equal to a second threshold, the candidate point is determined as the vanishing point.

9. The method according to claim 1, wherein the processing the image to obtain frame information of at least one of the three-dimensional units comprises: Cropping the image to obtain a cropped image including at least one of the three-dimensional units; Performing perspective transformation on the cropped image to obtain a corrected image, wherein the shape of the first surface of the stereoscopic unit in the corrected image is the same as the shape of the first surface in the three-dimensional structure of the stereoscopic unit; The corrected image is processed to obtain frame information of at least one of the three-dimensional units.

10. The method according to claim 9, wherein processing the rectified image to determine the frame information of at least one of the stereo units comprises: Extracting features of the corrected image through a feature extraction layer of a border detection model to obtain feature information of the corrected image; The frame recognition layer of the frame detection model processes the feature information to determine the corrected image wherein the target area is used to indicate the area in the corrected image where the stereo unit is displayed; Determine at least one unit feature information according to the at least one target area and the feature information through the pooling layer of the border detection model; wherein the unit feature information is used to characterize the feature information of the stereo unit; The linear mapping layer of the frame detection model is used to linearly map each unit feature information to obtain frame information of at least one of the three-dimensional units.

11. The method according to claim 9, wherein the step of performing perspective transformation on the cropped image to obtain a rectified image comprises: Determine the regional information of the corrected image according to the regional information of the cropped image; wherein the regional information of the cropped image is used to characterize the positions of the four vertices of the cropped image in the image, and the regional information of the corrected image is used to characterize the positions of the four vertices of the corrected image in the image; Calculating a transformation matrix according to the region information of the cropped image and the region information of the rectified image; wherein the transformation matrix is ​​used to transform pixel points in the cropped image into pixel points in the rectified image; For each pixel in the cropped image, determining a position of the pixel in the rectified image according to the position of the pixel in the cropped image and the transformation matrix; According to the position of each pixel point in the corrected image, the pixel points are arranged to obtain the corrected image.

12. The method according to claim 9, wherein cropping the image to obtain a cropped image including at least one of the stereoscopic units comprises: Acquire clipping region information for the image, the clipping region information including positions of four clipping vertices of the clipping image in the image; In the image, other regions except the clipping region obtained by connecting the four clipping vertices are removed to obtain the clipping image.

13. A device for determining three-dimensional information, the device comprising: An image acquisition module, used to acquire an image obtained by photographing the outer surface of the photographed object; wherein the outer surface of the photographed object has n protruding three-dimensional units, the three-dimensional unit refers to a solid unit with a three-dimensional structure, and n is a positive integer; a frame determination module, configured to process the image to obtain frame information of at least one of the three-dimensional units; wherein the frame information is used to locate a first surface and a second surface of the three-dimensional unit in the image, the first surface is parallel to the second surface, and the first surface or the second surface is within the outer surface; A three-dimensional determination module is used to determine the three-dimensional information of the three-dimensional unit according to the frame information of the three-dimensional unit; wherein the three-dimensional information is used to characterize the position of the three-dimensional unit in the image and the degree of protrusion of the three-dimensional unit in a direction perpendicular to the outer surface.

14. A computer device, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method for determining three-dimensional information according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein a computer program is stored in the storage medium, and the computer program is loaded and executed by a processor to implement the method for determining three-dimensional information according to any one of claims 1 to 12. 16 . A computer program product comprising a computer program, which, when executed on a computer, enables the computer to execute the method for determining three-dimensional information according to claim 1 .

Images