Target size measurement methods, electronic devices and their media

Abstract

This application relates to the fields of computer vision and artificial intelligence technology, and provides a method for measuring the size of a target. The method includes: acquiring two-dimensional and three-dimensional information images of a real scene, including the real plane where the target is located, wherein the pixels in the two-dimensional and three-dimensional information images have a one-to-one correspondence in position; obtaining the position information of the plane from the two-dimensional information image, and obtaining the three-dimensional pixel information of the corresponding pixels in the three-dimensional information image; converting the obtained three-dimensional pixel information into a point cloud, and generating a virtual plane corresponding to the real plane in virtual space based on the converted point cloud; and measuring the size of the target using the virtual plane as a reference plane. This method eliminates the need for users to manually select a measurement starting point or a measurement reference plane, making the size measurement more intelligent and solving the problem of target size deviation caused by the inaccuracy of the user-selected measurement reference plane.

Description

Technical Field

[0001] This application belongs to the field of computer vision and artificial intelligence technology, and in particular relates to a method for measuring the size of a target, an electronic device and a medium thereof. Background Technology

[0002] Augmented reality (AR) technology combines image recognition and localization, computer graphics, and visualization techniques to generate virtual objects and accurately integrate them into the real-world environment, providing users with a realistic sensory experience. AR technology has been widely applied in numerous fields, such as industrial maintenance, medicine, historical site restoration and digital cultural heritage preservation, and mobile application development. In mobile application development, AR technology can be used to measure the length and height of objects, such as the height of people or animals.

[0003] However, currently, when users use AR technology to measure the length, height, etc. of the target, they always need to manually select the measurement starting point or measurement reference plane, which makes the AR measurement process cumbersome, the measurement results inaccurate, and affects the user experience. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method for measuring the size of a target. In this method, the mobile phone fits the plane in the AR scene where the target is located based on the depth image of the plane in the real scene, and directly uses this plane as the reference plane for size measurement to measure the height of the target. This eliminates the need for the user to manually select the measurement starting point or the measurement reference plane, which not only simplifies the size measurement operation and makes the size measurement more intelligent, but also solves the problem of deviation in the size measurement of the target caused by the user's selection of an inaccurate measurement reference plane.

[0005] In a first aspect, embodiments of this application provide a method for measuring the size of a target, comprising: acquiring a two-dimensional information image and a three-dimensional information image of a real plane in a real scene, including the plane where the target to be measured is located, wherein the pixels in the two-dimensional information image and the three-dimensional information image have a one-to-one correspondence in position; obtaining the position information of the plane based on the two-dimensional information image, and obtaining the three-dimensional pixel information of the pixels corresponding to the position information in the three-dimensional information image; converting the obtained three-dimensional pixel information into a point cloud, and generating a virtual plane corresponding to the real plane in a virtual space based on the converted point cloud; and measuring the size of the target to be measured using the virtual plane as a reference surface.

[0006] Optionally, the two-dimensional information image represents the two-dimensional features of the target under test, wherein the two-dimensional features include one or more of color features, grayscale features, and texture features, while the three-dimensional information image represents the three-dimensional features of the target under test, wherein the three-dimensional features include the spatial depth value of the target under test, which refers to the distance between any part (or any point) of the target under test and the camera that captured the target under test.

[0007] Since there is a one-to-one correspondence between the pixels of the two-dimensional information image and its corresponding three-dimensional information image, the three-dimensional pixel information corresponding to the plane in the three-dimensional information image can be further determined by the real plane in the two-dimensional information image. The three-dimensional pixel information corresponding to the real plane is converted into the corresponding point cloud by the intrinsic parameter matrix and mapping formula of the camera (the camera in this application). Finally, the virtual plane corresponding to the real plane is fitted in the virtual space based on the point cloud of the real plane, and the size of the target to be measured is measured by using the virtual plane as the reference plane for size measurement.

[0008] Optionally, in one possible implementation, the target to be measured can be an object perpendicular to the ground, such as a person, animal, or building. Correspondingly, the size of the target to be measured can be the height of a person, the height of an animal, the height and width of a building, etc. It should be understood that if the target to be measured is an object fixed to a wall at one end, and the distance between the end of the target to be measured and the end of the target to be measured away from the wall is to be measured, the measurement method is the same as the measurement method for objects perpendicular to the ground described above. At that time, the electronic device will fit a virtual space "wall" corresponding to the wall based on the three-dimensional pixel information of the wall in the real scene and its corresponding point cloud, and use the "wall" of the virtual space as a reference surface to measure the size of the target to be measured.

[0009] Alternatively, in one possible implementation, the virtual space includes an augmented reality (AR) scene, the two-dimensional information image includes a color space image, the color space image includes an RGB image or a YUV image, and the three-dimensional information image includes a depth image.

[0010] Taking virtual space as the AR scene, two-dimensional information image as RGB image, and three-dimensional information image as depth image as an example, through the above method, when an electronic device, such as a mobile phone, measures the size of the target, it can automatically generate a virtual "ground" in the AR scene based on the RGB image and depth image of a plane, such as the ground, acquired by the camera. Then, the size of the target is measured using this virtual "ground". In this process, the user does not need to manually select the measurement starting point or measurement reference surface, which simplifies the operation of measuring the size of the target and makes the whole operation more intelligent. At the same time, it avoids the error that occurs when measuring the size of the target due to the inaccuracy of the measurement starting point or measurement reference surface manually selected by the user.

[0011] Optionally, the above-mentioned two-dimensional and three-dimensional information images can also be acquired by simultaneously acquiring two-dimensional and three-dimensional information images, including the actual plane in which the target is located, through a depth-sensing camera of an electronic device. The depth-sensing camera of the electronic device includes a time-of-flight (TOF) camera.

[0012] In one possible implementation of the first aspect, the electronic device can also utilize a semantic segmentation model to identify a plane in the two-dimensional information image, and then obtain the plane's position information. This position information can be obtained by determining the plane's region or its contour. For example, assuming the obtained information is the position of the ground in a real scene, the electronic device will utilize the semantic segmentation model to identify the ground region or its contour in the two-dimensional information image, and then determine the three-dimensional pixel information corresponding to the ground portion in the three-dimensional information image based on the ground region or its contour.

[0013] In one possible implementation of the first aspect, the semantic segmentation model is a fully convolutional neural network (FCN) model. Optionally, the semantic segmentation model can also be other neural network models; this application does not limit the type of neural network model on which the semantic segmentation model is based.

[0014] In one possible implementation of the first aspect, the acquired 3D pixel information is converted into a point cloud, and a virtual plane corresponding to the real plane in the virtual space is generated based on the converted point cloud, including: generating multiple sub-virtual planes corresponding to the real plane in the virtual space based on the converted point cloud; determining the plane confidence of each virtual plane in the multiple sub-virtual planes, and generating a virtual plane based on some of the sub-virtual planes in the multiple sub-virtual planes, wherein the plane confidence represents the ratio between the number of points in the point cloud of each sub-virtual plane and the number of points in the point cloud corresponding to the real plane.

[0015] Since a real plane may not be perfectly flat, when generating a virtual plane in the virtual space based on the point cloud corresponding to the real plane, one or more sub-virtual planes corresponding to the real plane may be generated. These sub-virtual planes are then fitted based on their plane confidence scores to generate a larger virtual plane, with the largest virtual plane being used as the virtual plane corresponding to the real plane. The specific fitting process can be found in the detailed description of plane fitting in the specific embodiments below, and will not be elaborated upon here.

[0016] When a virtual plane is generated in virtual space and used as a reference plane to measure the size of the target object, the specific measurement process can be as follows:

[0017] In one possible implementation, taking a person as an example, the above measurement method further includes: acquiring a head image of the person; determining the vertex of the person's head from the head image; and taking the distance between the vertex of the head and the virtual plane as the person's height.

[0018] In another possible implementation, to improve measurement accuracy, a 3D face recognition method can be used to determine the vertex of the person's head. Specifically, the head image described above is a 3D head image; facial feature points are then identified from this image using 3D face recognition; and the vertex of the head is determined based on these facial feature points. Finally, the distance between the vertex of the person's head and the virtual plane is taken as the person's height.

[0019] Secondly, embodiments of this application provide a computer-readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform a size measurement method for a target as described in any of the first aspects.

[0020] Thirdly, embodiments of this application provide an electronic device, including: a camera; an inertial measurement unit (IMU); one or more processors; one or more memories; a module with multiple applications installed; the memory stores one or more programs, the one or more programs including instructions that, when executed by the electronic device, cause the electronic device to perform a method for measuring the size of a target as described in any of the first aspects.

[0021] Fourthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes a processor to execute a size measurement method for the objective as described in any of the first aspects.

[0022] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a user interface (UI) for AR measurement using an AR measurement APP on a mobile phone, provided in an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of an example height calculation method provided in an embodiment of this application;

[0026] Figure 3 This is a UI change diagram of a height measurement using an AR measurement APP on a mobile phone (i.e., AR measurement APP) provided in an embodiment of this application;

[0027] Figure 4 This is an example UI diagram provided in an embodiment of this application;

[0028] Figure 5 This is an example UI diagram provided in an embodiment of this application;

[0029] Figure 6 This is an example UI diagram provided in an embodiment of this application;

[0030] Figure 7 This is an example UI diagram provided in an embodiment of this application;

[0031] Figure 8 This is an example UI diagram provided in an embodiment of this application;

[0032] Figure 9 This is an example UI diagram provided in an embodiment of this application;

[0033] Figure 10 This is a schematic diagram of a mobile phone hardware structure provided in an embodiment of this application;

[0034] Figure 11(a) is a schematic flowchart of a human height measurement method provided in an embodiment of this application;

[0035] Figure 11(b) is a schematic diagram of a plane fitting process provided in an embodiment of this application;

[0036] Figure 12 This is a schematic diagram of an example height calculation method provided in an embodiment of this application;

[0037] Figure 13This is a schematic diagram illustrating an example of using an image recognition method to identify the outline of a human head, as provided in an embodiment of this application.

[0038] Figure 14(a) is a flowchart of a method for determining the vertex of a person's head and calculating a person's height using three-dimensional face recognition technology, provided in an embodiment of this application.

[0039] Figure 14(b) is a schematic diagram of the process of generating a face mesh using three-dimensional face recognition technology according to an embodiment of this application;

[0040] Figure 15 This is a schematic diagram of facial feature points of a person provided in an embodiment of this application;

[0041] Figure 16 This is a schematic diagram of the software framework of a mobile phone provided in an embodiment of this application. Detailed Implementation

[0042] The use of "attributes" in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0043] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0044] In the realm of mobile device applications, mobile AR technology is commonly used to measure the length and height of objects, such as the height of people or animals. However, regardless of the specific application, when users perform AR measurements using their phones, they need to manually select a starting point for the measurement, which is cumbersome. Specifically, Figure 1 Figures (a) to (d) show UI diagrams of AR measurement using an AR measurement app on a mobile phone.

[0045] like Figure 1 As shown in Figure (a), the user can select the "Height Measurement" button 102 in the "Measurement Mode Options" 101. In response to the user's operation, the mobile phone 10 enters the height measurement mode. Subsequently, the display interface UI 100 of the AR measurement APP prompts the user to move the mobile phone 10 to find the ground so as to identify the ground and fit the ground in the AR scene.

[0046] like Figure 1As shown in Figure (b), once the AR scene is successfully established, UI 100 will display "Ground recognition successful" to indicate to the user that height measurement can now be performed. At this point, in order to measure height, the user can first select the measurement starting point P3' on the sole of the person being measured.

[0047] Then, as Figure 1 As shown in Figure (c), the mobile phone 10 will construct a measurement reference plane β with the measurement starting point selected by the user, and prompt the user to move the mobile phone 10 upward to recognize the face.

[0048] Finally, as Figure 1 As shown in Figure (d), the mobile phone 10 identifies the face of the person being tested, determines the vertex of the head of the person being tested, and displays the height of the person being tested, that is, the distance of the vertex of the head from the measurement reference plane β, "1.8m".

[0049] Correspondingly, the specific process for calculating the height of the person being measured is as follows: Figure 2 As shown, it includes:

[0050] The mobile phone 10 uses the starting point P3' manually selected by the user as the reference to fit the measurement reference plane β, and then calculates the distance from the reference plane β to the top of the head of the person being measured, P1, i.e., P2'P1, and uses this distance as the height of the person being measured.

[0051] However, it is clear that the height of the person being tested should be the distance from the top of the head P1 to the ground α in the AR scene fitted by the mobile phone 10, i.e., P1P2.

[0052] Therefore, if the height is measured by fitting a reference plane to the measurement starting point manually selected by the user, the reference plane and the actual ground in the AR scene will not be on the same horizontal plane, which will lead to errors in the measurement of the height of the person being measured.

[0053] Furthermore, if the plane fitted by the phone 10 based on the user-selected measurement starting point and two surrounding points is tilted, the measurement error for the person's height will be even greater. For example, as... Figure 2 As shown in Figure (b), if the user selects P3" as the starting point of the measurement, the reference plane ω fitted by the mobile phone 10 based on the starting point P3" has an angle with the actual ground α in the AR scene. The height P1P2" of the person to be measured obtained based on the reference plane ω is much different from the actual height P1P2 of the person to be measured.

[0054] To address the aforementioned technical problems, this application provides a method for measuring the size of a target. In this method, the user does not need to manually select the starting point for measurement. As the user slowly moves the mobile phone 10 to find the ground, the phone 10, through a trained semantic segmentation model, can accurately identify the ground in the real-world scene and fit the ground in the AR scene based on the depth image of that ground. Then, the phone 10 directly uses the ground in the AR scene as a reference surface to measure the size of the target.

[0055] Furthermore, when the target is a person and their height is measured, this application also utilizes 3D (dimension) face recognition technology to identify the face and generate a 3D face mesh model. Unlike 2D images, which can only identify the outline of a person, a 3D face mesh is a mask-like model that can directly and closely fit the real face. Since the tip of the nose is the most prominent feature in a 3D face mesh, it is used as a feature point to determine the vertex of the person's head. The distance from the vertex of the head to the ground in the AR scene is then calculated, and this distance is used as the height of the person being measured, thus achieving a more accurate measurement of the person's height.

[0056] The following will be combined with the appendix Figures 3 to 9 The method for measuring the dimensions of the target object in this application is described in further detail.

[0057] For ease of explanation, this document will use the measurement of a person's height using a mobile phone 10 as an example to describe the solution of this application. However, it should be understood that the solution of this application can be applied not only to mobile phones 10, but also to electronic devices such as tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, and ultra-mobile personal computers (UMPCs). The embodiments of this application do not impose any limitations on the specific types of electronic devices that can implement the solution of this application.

[0058] Figure 3 Figures (a) to (c) are UI change diagrams of a height measurement using an AR measurement APP on a mobile phone (i.e., AR measurement APP) provided in an embodiment of this application.

[0059] like Figure 3 As shown in Figure (a), users can open the AR Measurement App's display interface (UI 300) by clicking the AR Measurement App icon on the phone's home screen to measure the target. When the user clicks "AR Measurement" 301, UI 300 will display as shown in Figure (a). Figure 3The startup interface for "AR Measurement" is shown in Figure (b).

[0060] After the startup screen ends, the user will enter the main interface of the AR measurement application, such as... Figure 3 As shown in Figure (c), the "Measurement Mode Options Bar" 302 of UI 300 displays "Length Measurement" mode 302a, "Area Measurement" mode 302b, and "Volume Measurement" mode 302c, including "Height Measurement" mode 302d. The function of each measurement mode is also displayed above it. Taking "Length Measurement" mode 302a as an example, when the user presses and holds the "Length Measurement" button 302a, the corresponding function "Measure the length of an object" 303a is displayed as a floating window above the mode. Simultaneously, the desktop length measured using this mode, "1.8m," is displayed in area 303b, thus indicating the functions and effects of each measurement mode.

[0061] Optionally, the aforementioned startup screen may be displayed once when the user first installs and opens the "AR Measurement" application, or it may be displayed every time the user opens the "AR Measurement" application; this application does not limit this.

[0062] When the user clicks the "Height Measurement" button (302d), UI 300 redirects to the interface corresponding to the height measurement mode, i.e., UI 400. Specifically, as follows... Figure 4 As shown in Figure (a), when the user selects "Height Measurement Mode", the 401 area of ​​UI 400 will display "Automatic Measurement Mode is On", and the phone 10 will prompt the user to "Slowly move the device and find the ground".

[0063] Once ground recognition is successful, such as Figure 4 As shown in Figure (b), area 401 of UI 400 will display "Ground recognition successful," and phone 10 will prompt the user to "Slowly move the device to find the face." At this time, the user can move phone 10 upwards so that phone 10 can recognize the face of the person being tested (see reference). Figure 4 (See Figure (c)). When the phone 10 recognizes a face, it will display the height of the person being tested as "1.8m" in area 405 of UI 400.

[0064] In one possible implementation, when a user wants to reselect the measurement starting point, they can reselect the measurement test point through the undo operation or the add measurement starting point operation in the function option bar 404.

[0065] Optionally, the height of the person being measured can be displayed in any position on UI 400 that can be seen by the user; this application does not impose any restrictions on this.

[0066] Optionally, the unit of the height of the person being measured can be set by the user by clicking the settings button 402. The unit of the height of the person being measured can be a metric unit such as m or cm, or an imperial unit. This application does not impose any restrictions on this.

[0067] Furthermore, in another possible implementation, the user can also take a screenshot of the current interface displaying the height of the person being measured using the camera function within option 404 (see reference). Figure 4 (b) in the image, and save it as an image.

[0068] It can be seen that the height measurement procedure in this application differs from the height measurement procedure in traditional methods. Figure 4 In the height measurement process shown, after the user selects the height measurement function and enters the height measurement mode, the user only needs to move the mobile phone 10 so that the mobile phone 10 can build an AR scene and recognize the face, without the user needing to select the measurement starting point. This makes the height measurement process more intelligent and improves the user experience.

[0069] In some embodiments of this application, the mobile phone 10 can also measure the height of multiple individuals simultaneously. Specifically, when the user clicks the "Height Measurement" mode 302d button to enter the height measurement mode (see reference...), Figure 4 (Figure (a) in the image) When the mobile phone 10 successfully recognizes the ground, as shown in Figure (a), Figure 4 As shown in Figure (b), area 401 of UI 400 will display "Ground recognition successful". Simultaneously, phone 10 will prompt the user to "Slowly move the device to find the face". The user follows the prompt and slowly moves phone 10, as shown... Figure 5 As shown in Figure (a), mobile phone 10 recognizes three faces. Then, as... Figure 5 As shown in Figure (b), UI 300 simultaneously displays the heights of the three subjects, “1.8m”, “1.9m”, and “1.75m”, in area 501.

[0070] Optionally, the phone 10 can also display the height difference of the three individuals being measured in area 501, increasing the fun of height measurement. For example, as Figure 5 As shown in Figure (c), area 501 can also display the height difference between the three test subjects and the tallest test subject, which is “0.3m” and “0.35m”.

[0071] Optionally, the aforementioned height difference can be the difference between the height of each test subject and the height of the tallest test subject among all test subjects, or the difference between the height of each test subject and the height of the shortest test subject among all test subjects, or the height difference between adjacent test subjects, etc. This application does not impose any restrictions on the calculation method of this height difference.

[0072] In some other embodiments of this application, when multiple people are present on the current screen of the mobile phone 10, the user can select one or more specific people to measure their height, thereby increasing the intelligence of the height measurement.

[0073] Specifically, when a user clicks the "Height Measurement" mode button on the 302d device to enter the height measurement mode (see reference...), Figure 4 (Figure (a) in the image) When the mobile phone 10 successfully recognizes the ground, as shown in Figure (a), Figure 4 As shown in Figure (b), area 401 of UI 400 will display "Ground recognition successful". Simultaneously, phone 10 will prompt the user to "Slowly move the device to find the face". The user follows the prompt and slowly moves phone 10, as shown... Figure 5 As shown in Figure (a), after the mobile phone 10 recognizes three faces, the user can select the middle person as the person to be measured for height measurement according to their needs.

[0074] like Figure 6 As shown in Figure (a), users can move the "selection box" 601 left, right, up, and down to select the person whose height they want to measure.

[0075] Optionally, in some embodiments of this application, the "selection box" 601 may also be displayed when the user selects a specific target to be tested, so as to prompt the user about the currently selected target to be tested.

[0076] like Figure 6 As shown in Figure (b), when the user selects the person in the middle as the test subject, the mobile phone 10 prompts the user to "slowly move the device to find the face". The user follows the guidance so that the mobile phone 10 can recognize the face of the test subject selected by the user.

[0077] Finally, as Figure 6 As shown in Figure (c), the height of the person being measured is displayed as "1.9m" on the UI 600.

[0078] In other embodiments of this application, when the distance between the ceiling and the floor is fixed, such as indoors, the mobile phone 10 can also determine the height of the person being measured by recognizing the ceiling and the face of the person being measured, and then using the known distance between the ceiling and the floor. Specifically, when the user clicks the "Height Measurement" mode 302d button to enter the height measurement mode (see reference...), Figure 4 (Figure (a) in the middle), as shown Figure 7 As shown in Figure (a), the UI 700 of the mobile phone 10 will display "Automatic measurement mode is enabled" and prompt the user to "Slowly move the mobile device to find the ceiling". The user follows the guidance to move the mobile phone 10 to find the ceiling so that the mobile phone can recognize the ceiling.

[0079] Once the phone successfully identifies the ceiling, area 701 of UI 700 will display "Ceiling recognition successful". Figure 7 As shown in Figure (b), at this time, the mobile phone 10 will prompt the user to "slowly move the mobile device to find the face". The user follows the guidance and moves the mobile phone 10 to find the face so that the mobile phone can recognize the face of the person being tested.

[0080] Optionally, to more clearly identify the user's face, the camera 193 of the phone 10 will increase its focal length and narrow its field of view to magnify the face of the person being tested, such as... Figure 7 As shown in Figure (c), it can be seen that the face of the test subject displayed by UI 700 is larger than... Figure 7 The face of the person being tested is shown in (a) and (b) of the image in the UI 700.

[0081] It should be understood that the field of view can be expressed in terms of the field of view angle, which is inversely proportional to the focal length of the camera on the phone. The larger the focal length of the camera, the smaller the field of view angle, and the smaller the range of objects that can be displayed; conversely, the smaller the focal length of the camera, the larger the field of view angle, and the larger the range of objects that can be displayed.

[0082] When the phone successfully recognizes the face of the person being tested, the phone will display the height of the person being tested as "1.8m" in area 702 of UI 700.

[0083] Additionally, when there are multiple people spaced far apart in the area where the phone 10 is located, the display screen 194 of the phone 10 may not be able to display all of them simultaneously; or, when the phone 10 has already displayed the height of some of the people to be measured, and then other people enter the frame, and the user does not want to repeat the measurement... Figure 4 Figure (a) in the middle Figure 4 In step (c) of the diagram, the user can directly move the phone 10 in the current mode to recognize the faces of other people so that the phone 10 can continue to measure the height of other people and display their height.

[0084] Specifically, Figure 8 This is an example UI diagram provided by some embodiments of this application.

[0085] like Figure 8 As shown in Figure (a), at this point, the mobile phone 10 has completed the measurement of the height of the person being measured, Tom, and displays the height of the person being measured, Tom, as "1.8m".

[0086] However, Joey, the test subject, is also present in the room at the same time. Since Joey and Tom are quite far apart, the current screen of the phone cannot accommodate both of them simultaneously. Therefore, if... Figure 8As shown in Figure (a), the user moves phone 10 relatively away from the test subject Tom, while simultaneously moving phone 10 to the left to locate the test subject Joey so that phone 10 can recognize Joey's face, until phone 10 can simultaneously accommodate both the test subject Tom and the test subject Joey (see reference). Figure 8 (Figure (b) in the middle).

[0087] Then, as Figure 8 As shown in Figure (c), after the mobile phone 10 recognizes the face of the test subject Joey, it will display Joey's height as "1.9m".

[0088] Optionally, to make it easier for users to visually see the heights of test subjects Tom and Joey, such as... Figure 8 As shown in Figure (d), the phone 10 can record the heights of Tom and Joey in panoramic mode. At this time, area 801 of UI 800 displays "Panorama" and shows the heights of the two people in the panoramic image. At this time, the user can also swipe left and right on UI 800 to display other details in the panoramic photo (such as tables, doors, and other objects recorded in the panoramic image).

[0089] It should be understood that panorama mode is a mode in which more objects are included in the frame when the phone is moved or rotated during shooting. The camera automatically takes multiple photos during the movement or rotation, and then stitches them into a panoramic image using the phone's built-in software.

[0090] Optionally, when there are many test subjects and the phone 10 cannot display all of them in portrait mode, such as Figure 9 As shown, users can rotate their phone (phone 10) to landscape mode and measure the height of the person being measured. The specific height measurement operation method is the same as above. Figures 3 to 8 The method is the same as in the previous section, so it will not be repeated here.

[0091] The above embodiments, for the purpose of facilitating the description of the operation method for measuring the size of the target object of this application, use a human as an example. It should be understood that the applicable objects of this application include, but are not limited to, humans, and the size measurement method of the target object of this application can also be applied to other objects. For example, objects with feature points, such as animals, plants, and buildings. Feature points refer to points or sets of points that can be identified based on a neural network model and can characterize the shape, color, height, etc., of an object. For example, the corner points of a building in an image can characterize the shape and edges of the building; the tip of a human nose and the top of a human head can characterize the height of the human body; the top of a plant can characterize the height of the plant, and so on.

[0092] To understand the implementation process of this application, the following text will still use the target being measured as a person and the height of the person being measured as an example, combined with the appendix. Figures 10 to 16The scheme of this application will be further described.

[0093] Figure 10 This is a schematic diagram of the hardware structure of a mobile phone 10 according to an embodiment of this application. Figure 10 As shown in Figure (a), the mobile phone 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, cameras 193 and 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0094] The sensor module 180 may include a pressure sensor 180A, an inertial measurement unit 180B (including a gyroscope sensor 1801B and an accelerometer sensor 1802B), a magnetic sensor 180C, a barometric pressure sensor 180D, a distance sensor 180E, a proximity light sensor 180F, a fingerprint sensor 180G, a temperature sensor 180H, a touch sensor 180J, an ambient light sensor 180K, a bone conduction sensor 180L, etc.

[0095] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the mobile phone 10. In other embodiments of this application, the mobile phone 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements.

[0096] The components illustrated can be implemented in hardware, software, or a combination of both. Processor 110 may include one or more processing units, such as: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units can be independent devices or integrated into one or more processors.

[0097] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or is reusing. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. In some embodiments of this application, the processor 110 is responsible for building an AR scene based on the depth image of the ground acquired by the mobile phone 10.

[0098] The mobile phone 10 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. In some embodiments of this application, a graphics rendering tool (Graphics Kit) can be used to render a measurement reference plane in an AR scene.

[0099] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 10 may include one or N displays 194, where N is a positive integer greater than 1.

[0100] In this embodiment, the display screen 194 can be used to display the various apps and function options mentioned above, and in response to user operations, to measure the height of the person to be measured and display the measured height value on the display screen 194. The specific measurement method can be found in the relevant descriptions of subsequent embodiments, and will not be repeated here.

[0101] Mobile phone 10 can perform shooting (or photo taking) functions through ISP, camera 193, video codec, GPU, display 194, and application processor. In some embodiments of this application, the user can take a screenshot of the interface displaying the height of the person to be measured and save it. This will be described in detail below.

[0102] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the photosensitive element of the camera 193. The light signal is converted into an electrical signal, and the photosensitive element of the camera 193 transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193. The camera 193 is used to capture still images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, and other formats. In some embodiments, the mobile phone 10 may include one or N cameras 193, where N is a positive integer greater than 1.

[0103] For example, in some embodiments of this application, such as Figure 10 As shown in Figure (b), the mobile phone 10 may include five rear cameras 193, namely a multi-reflection periscope telephoto camera 193, an ultra-sensing camera 193, a telephoto camera 193, a cine camera 193, and a depth-sensing camera 193. In some embodiments of this application, the mobile phone 10 acquires a depth image of the ground in the space where the mobile phone 10 is located through the depth-sensing camera 193, and establishes an AR scene and fits the ground in the AR scene based on the acquired depth image. The specific method will be described in detail below.

[0104] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn. NPU enables intelligent cognitive applications on the mobile phone 10, such as image recognition, face recognition, speech recognition, and text understanding. In some embodiments of this application, the mobile phone 10 uses the NPU to recognize the face of the person being tested, thereby determining the vertex of the person's head. In other embodiments of this application, the mobile phone 10 uses the NPU to train a semantic segmentation model based on a semantic segmentation database. This allows the mobile phone 10 to more accurately identify the ground as the user moves it, and based on this, construct a more accurate AR scene, enabling the mobile phone 10 to accurately fit the ground in the measured scene. This will be described in detail below.

[0105] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Mobile phone 10 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, mobile phone 10 detects the intensity of the touch operation based on pressure sensor 180A. Mobile phone 10 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed. In other embodiments of this application, the mobile phone 10 can execute corresponding instructions based on specific operations performed by the user on the display screen 194. For example, if the user clicks the "Height Measurement" button, the mobile phone 10 will display the current page as shown below. Figures 3 to 9 The height measurement interface is shown below. A detailed description follows.

[0106] The gyroscope sensor 1801B can be used to determine the motion attitude of the mobile phone 10. In some embodiments, the angular velocity of the mobile phone 10 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 1801B. The gyroscope sensor 1801B can be used for image stabilization.

[0107] For example, when the shutter button is pressed, the gyroscope sensor 1801B detects the angle of the phone 10's shake, calculates the distance the lens module needs to compensate based on the angle, and allows the lens to counteract the phone 10's shake through reverse movement, thus achieving image stabilization. The gyroscope sensor 1801B can also be used in navigation and motion-sensing gaming scenarios.

[0108] The 1802B accelerometer sensor can detect the magnitude of acceleration in various directions (typically three axes) of the mobile phone 10. When the mobile phone 10 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the mobile phone 10, and can be applied to applications such as landscape / portrait switching and pedometers.

[0109] In some embodiments of this application, the gyroscope sensor 1801B and the accelerometer sensor 1802B constitute an inertial measurement unit (IMU) 180B. The IMU unit 108B is used to acquire the acceleration and angular velocity of the camera 193 of the mobile phone 10 at various times, so as to determine the pose information of the camera 193 at various times. This allows the mobile phone 10 to perform planar fitting based on the pose information of the camera 193 at various times. The pose information includes the three-dimensional coordinates and orientation of the camera 193 at various times. The pose information of the camera 193 can represent the positional changes of the camera 193. The mobile phone 10 will fit the ground in the AR scene based on the depth image of the space where the subject is located, acquired by the camera 193 at different positions, so that the ground in the AR scene is closer to the ground in the real scene. This will be described in detail below.

[0110] It should be understood that, unless otherwise specified, the coordinates and coordinate systems mentioned in this article refer to world coordinates and the world coordinate system.

[0111] In combination with the above Figures 3 to 9 The UI diagram is shown below. The specific implementation process of the target size measurement method of this application is described below.

[0112] Typically, when fitting a plane in an AR scene, the mobile phone 10 needs to obtain the depth image of the plane in the real scene, then convert the coordinates of each pixel in the depth image in the image coordinate system to the world coordinate system to obtain the point cloud corresponding to the depth image, and fit the plane in the AR scene based on the point cloud.

[0113] As mentioned earlier, in the scenario of AR measuring the height of a person being measured, the mobile phone 10 does not perfectly fit the depth image of the ground in the real scene when fitting the ground in the AR scene. Specifically, when the user uses the "AR Measurement" app and moves the mobile phone 10 to find the ground, the plane fitted by the mobile phone 10 is not the ground in the AR scene. Therefore, when measuring the height of the person being measured, the mobile phone 10 needs the user to select a measurement starting point at the bottom of the person being measured (refer to Figure 1(b)). Then, it performs plane fitting based on the measurement starting point selected by the user and uses this plane as the measurement reference plane for measuring the height of the person being measured, that is, the ground in the AR scene, and measures the height of the person being measured. Thus, when the measurement starting point selected by the user is not a point in the ground of the real scene, the ground in the AR scene fitted by the mobile phone 10 based on the measurement starting point will be inaccurate, resulting in the height measurement error mentioned earlier.

[0114] To avoid the aforementioned situation, in the target size measurement method of this application, the mobile phone 10 will incorporate semantic segmentation technology to train a semantic segmentation model for recognizing plane types in real-world scenes. Then, when the user "slowly moves the mobile phone 10 to find the ground" (see reference...),... Figure 4 When (a) is in the picture, the mobile phone 10 will continuously acquire the color system (RGB) image corresponding to the camera 193 in different poses and the depth image corresponding to the RGB image through the camera 193, and use the trained semantic segmentation model to identify the acquired RGB image, determine the ground in the RGB image, and then directly fit the plane in the AR scene with the point cloud of the ground in the RGB image and the ground in the depth image corresponding to the RGB image.

[0115] In this way, the phone can directly use the plane as the ground in the AR scene, that is, the reference plane for measuring the height of the person being measured, so as to make the AR measurement operation simpler and more intelligent, and at the same time avoid the errors that occur in the measurement of the height of the person being measured in existing technologies.

[0116] Figure 11(a) is a schematic flowchart of a human height measurement method provided in some embodiments of this application.

[0117] As shown in Figure 11(a), method 1100 includes:

[0118] 1101: Mobile phone 10 acquires two-dimensional information images of the space in which a person is located in a real scene and three-dimensional information images corresponding to the two-dimensional information images through depth sensing camera 193.

[0119] Two-dimensional information images reflect the two-dimensional features of objects within the image, while three-dimensional information images reflect the three-dimensional features of objects within the image.

[0120] In one possible implementation of this application, the two-dimensional information image is an RGB image, also known as a color image or color system image, which is a common RGB three-channel color image. An RGB image reflects the two-dimensional features of objects in the image, such as color, grayscale, brightness, and sharpness. The pixels of an RGB image are RGB values, ranging from 0 to 225. Optionally, the two-dimensional information image can also be an image in other color spaces, such as a YUV image, where "Y" represents the grayscale value (luma), "U" represents the hue (hue), and "V" represents the saturation (value).

[0121] Three-dimensional information images are depth images, which are images that use the distance from the image acquisition device to various points in the scene as pixel values ​​(i.e., three-dimensional pixel information). For example, in this application, the depth-sensing camera 193 uses the distance from the distance to various points in the scene where the person being tested is located as pixel values. Depth images reflect the geometry of the visible surface of an object.

[0122] In this application, the mobile phone 10 acquires RGB images and depth images, i.e., RGB-D images, through the depth-sensing camera 193. It should be understood that the RGB images acquired by the depth-sensing camera 193 and their corresponding depth images are registered, and therefore their pixels have a one-to-one correspondence. For example, taking the ground RGB image in this application as an example, the pixels of the depth image corresponding to the ground RGB image are the distance values ​​from the ground to the depth-sensing camera 193.

[0123] Optionally, in one possible implementation, a regular camera 193 can be used to acquire the aforementioned RGB image, while a depth sensor, such as a LiDAR, can be used to acquire the aforementioned depth image. Then, image registration is performed on the RGB image and the depth image to ensure a one-to-one correspondence between the pixels of the RGB image and the depth image, thus obtaining the final RGB-D image. It should be understood that the aforementioned image registration method can involve normalizing the pixel values ​​of the RGB image and the depth image separately, and then using feature points in the RGB image, such as color feature points and texture feature points, and feature points in the depth image for image alignment. For example, taking a human face image as an example, the feature points in the RGB image of the human face are likely to remain feature points in the depth image, such as the tip of the nose and the nostrils. Therefore, the RGB image and the depth image of the human face can be aligned based on the feature points that overlap with the feature points in the depth image of the human face to obtain the RGB-D image of the human face.

[0124] Optionally, in some embodiments of this application, the user follows the guidance of mobile phone 10 to "slowly move the device and identify the ground" (see reference). Figure 4 During the process shown in Figure (a), the depth-sensing camera 193 of the mobile phone 10 can acquire an RGB-D image stream of the space in which the person being tested is located in the real scene. It should be understood that the RGB-D image stream is a video stream composed of multiple consecutive RGB-D images.

[0125] 1102: Mobile phone 10 identifies the ground in the two-dimensional information image, then uses the coordinates of the ground in the two-dimensional information image to obtain the point cloud of the ground in the corresponding three-dimensional information image, and uses the obtained ground point cloud to fit the ground where the person is in the AR scene.

[0126] After acquiring the RGB-D image of the space where the test subject is located in a real scene, the mobile phone 10 will use the trained semantic segmentation model to identify the ground in the RGB image, and then use the point cloud data of the ground in the depth image corresponding to the RGB image to fit the ground in the AR scene.

[0127] It is understandable that both AR and VR scenes are specific applications in the field of visual technology, and both involve fitting virtual planes and constructing virtual spaces. Therefore, the height measurement method of this application can also be applied to VR scenes.

[0128] It is understood that the semantic segmentation model in this application can identify the types of various planes contained in the RGB image in step 1101. For example, the mobile phone 10 uses the semantic segmentation model to identify various types of planes such as walls, floors, and desktops in the RGB image. The specific functions and training process of the aforementioned semantic segmentation model will be described in detail below.

[0129] It can also be understood that a point cloud is a collection of points obtained after scanning a 3D scene using instruments such as a 3D coordinate measuring machine, a photographic scanner (i.e., camera 193 in this application), or a laser scanner. Each point in the collection has definite 3D coordinates and corresponds to its position in the depth image. The distribution of point clouds can characterize the shape and position of objects.

[0130] In this application, the mobile phone 10 can determine the point cloud corresponding to the depth image in the RGB-D image by combining the depth image in the RGB-D image acquired by the depth-sensing camera 193 with the camera intrinsic parameter matrix of the mobile phone 10. Then, the mobile phone 10 performs plane fitting on the denser points in the point cloud based on the density or sparseness of the point cloud to obtain the plane in the AR scene. The method by which the mobile phone 10 fits the plane in the AR scene based on the point cloud will be described in detail below.

[0131] 1103: The mobile phone 10 uses the ground in the AR scene as the measurement reference surface for measuring a person's height, and measures a person's height.

[0132] After the mobile phone 10 fits the ground in the AR scene using the method in step 1102, the mobile phone 10 will use this ground as the measurement reference plane for measuring the person's height. Specifically, the mobile phone 10 identifies the vertex P1 of the person's head, then calculates the distance from the vertex P1 of the person's head to the measurement reference plane, and displays the distance from the vertex P1 of the person's head to the measurement reference plane as the person's height on the UI of the mobile phone 10 (e.g., ...). Figure 4 (as shown in Figure (d)). Figure 12 This is a schematic diagram illustrating a method for calculating human height provided in some embodiments of this application.

[0133] like Figure 12 As shown, the mobile phone 10 uses plane α as the reference plane for height measurement, calculates the distance P1P2 from the vertex P1 of the human head identified by the human head to the aforementioned reference plane, and displays P1P2 as the human height on the mobile phone 10.

[0134] It can be seen that the above height measurement method differs from traditional height measurement methods, such as... Figure 2 As shown in Figure (a), the traditional height measurement method requires the user to select a measurement starting point P3', and then the mobile phone 10 establishes a plane β based on the user-selected measurement starting point P3', and calculates the distance P1P2' between the plane β and the user's head vertex P1. This method of measuring a person's height will have errors because the plane β is not the ground where the person is located in the AR scene. However, the height measurement method of this application does not require the user to select a measurement starting point. The mobile phone 10 will directly use the established plane α as the measurement reference to measure the person's height, avoiding the error caused by the plane β established by the mobile phone 10 based on the user-selected measurement starting point not being in the same position as the actual ground in the AR scene, thus improving the accuracy of height measurement.

[0135] Specifically, the mobile phone 10 can identify the vertex P1 of the subject's head using either two-dimensional image recognition technology or three-dimensional face recognition technology. Details will be provided below.

[0136] By following steps 1101 to 1103, users can achieve the following on mobile phone 10. Figures 4 to 9 The method for measuring human height described in the document.

[0137] To facilitate a more intuitive understanding of the implementation process of steps 1101-1103, the specific implementation details of steps 1101-1103 will be introduced below with reference to Figure 11(b).

[0138] As shown in Figure 11(b), corresponding to the above steps 1101-1102, steps 1101-1102 can be divided into two parts, namely plane generation 1110b and semantic segmentation 1120b.

[0139] The plane generation 1110b includes acquiring a depth image 1111b and determining the point cloud 1113b corresponding to the depth image based on the acquired depth image. At the same time, it acquires the acceleration and angular velocity of the mobile phone 10 1112b and determines the pose information 1114b of the camera 193 using the SLAM algorithm based on the acceleration and angular velocity of the mobile phone 10. Then, it combines the point cloud 1113b in the depth image and the pose information of the camera 193 to fit a plane 1115.

[0140] At the same time, in semantic segmentation 1120b, RGB images 1121b corresponding one-to-one with the aforementioned depth image pixels are also acquired through camera 193. Then, the acquired RGB images are identified using the trained semantic segmentation model 1122b to determine the type of each plane in the RGB images and generate semantic labels 1123b, which refer to the type of each plane.

[0141] Then, the mobile phone 10 adds the generated semantic tag 1123b to the fitted plane, and finally generates a plane with semantic tag 1123b. Optionally, in the height measurement method of this application, the plane with semantic tag 1123b is the ground in the AR scene labeled "ground".

[0142] It is understood that the above-mentioned acquisition of depth image 1111b corresponds to step 1101 in method 1100. The specific method of acquiring depth image can be referred to the description of step 1101. The method of generating point cloud 1113 corresponding to the depth image based on the acquired depth image is the same as the method of determining point cloud based on depth image in step 1102, and will not be repeated here.

[0143] The following section describes the process of training the semantic segmentation model, combining steps 1102 and semantic segmentation 1120b.

[0144] Specifically, the training process of the semantic segmentation model in step 1102 above is as follows:

[0145] The mobile phone 10 trains the aforementioned semantic segmentation model based on images with pixel-level labels from an existing semantic segmentation dataset, enabling the trained semantic segmentation model to recognize the ground in the RGB image acquired by the depth-sensing camera 193. Here, an image with pixel-level labels means that each pixel in the image has a corresponding type label; for example, some pixels correspond to the ground, some to a wall, some to a human body, and some to a sofa, etc.

[0146] Specifically, the mobile phone 10 uses images from a semantic dataset with preset pixel-level category labels as target data; then the mobile phone 10 inputs the target data into the semantic segmentation model to be trained, obtains the semantic segmentation result of the target data, and calculates the loss function of the semantic segmentation model based on the semantic segmentation result of the target data.

[0147] Alternatively, the formula for calculating the loss function can be...

[0148]

[0149] in, The result after segmentation using a semantic segmentation model. Preset pixel-level category labels for the target data. This is the loss function for the semantic segmentation model.

[0150] Then, the phone adjusts the parameters in the semantic segmentation model based on the result of the loss function, such as the weight values ​​of each layer of the neural network used in the semantic segmentation model, to reduce the result of the loss function so that the output of the semantic segmentation model is the same as or similar to the input result (i.e., the preset pixel-level category label). When the output of the semantic segmentation model is the same as or similar to the input result, the semantic segmentation model is considered to have completed training.

[0151] In some embodiments of this application, the completion of semantic segmentation training can be determined by comparing the result of the loss function with a preset threshold. When the result of the loss function is less than or equal to the preset threshold, the output of the semantic segmentation model is considered to be the same as or similar to the input result, i.e., the semantic segmentation model training is complete. The setting of the preset threshold is related to the neural network model and the loss function used. Neural network models generally have better performance, and the preset threshold can be set lower. This application does not limit the method of setting the preset threshold.

[0152] Optionally, the semantic segmentation dataset used to train the semantic segmentation model can be a 2D dataset such as VOC (pascalvisual object classes) or MS COCO (cityscapes micro soft common objects in context); or a 2.5D dataset such as NYU-D V2, SUN-3D, or SUN RGB-D; or a 3D dataset such as Stanford 2D-3D or Shape-Net Core. This application does not limit the dataset used to train the semantic segmentation model.

[0153] Optionally, the semantic segmentation model used for training can be based on neural network model architectures such as fully convolutional networks (FCNs), Seg-Net, U-Net, and Deep-Lab V1-V3. This application does not limit the type of neural network model architecture used for semantic segmentation training.

[0154] It is understandable that the semantic segmentation training process mainly involves segmenting RGB images. Therefore, during the training process described above, the mobile phone 10 can also acquire RGB images for training through a regular RGB camera 193.

[0155] Optionally, the semantic segmentation model training process described above can be performed on mobile phone 10 or on other electronic devices. When the semantic segmentation model is trained by other electronic devices, after the semantic segmentation model training is completed, the other electronic device will package the semantic segmentation model into a software development kit (SDK) file, send it to mobile phone 10, and install it on mobile phone 10. When mobile phone 10 needs the semantic segmentation model, mobile phone 10 uses the semantic segmentation model by calling the corresponding function of the semantic segmentation model. These other electronic devices can be laptops, desktop computers, cloud computers, tablets, or other electronic devices with model training capabilities; this application does not limit the form of these other electronic devices.

[0156] The process of plane fitting is described below in conjunction with step 1102 and plane fitting 1110.

[0157] Specifically, the plane fitting process includes:

[0158] 1) The mobile phone 10 converts the depth image of the space where the object to be tested is located into the corresponding point cloud 1113b.

[0159] As mentioned above, the pixel value of each point in the depth image of the space where the object to be measured is located is the distance value from that point to the camera 193 of the mobile phone 10. Based on the pixel value of each point in the depth image, the Z coordinate of each point in the corresponding point cloud of the depth image can be calculated. Then, the X coordinate and Y coordinate of the point can be calculated from the Z coordinate of each point in the point cloud.

[0160] Specifically, the mobile phone 10 uses camera intrinsic parameters and a mapping formula to convert the depth image of the space containing the object under test into a corresponding point cloud 1113b. Optionally, the specific mapping formula can be:

[0161] (one)

[0162] (two)

[0163] (three)

[0164] in, yes

[0165] This refers to the pixel value of the point in the i-th row and j-th column of the depth image of the space where the object under test is located, which is also the distance of that point from the camera 193 of the mobile phone 10.

[0166] camera.scale represents the ratio of the pixel value of a point in the depth image of the space where the object being measured is located to the actual physical distance (in millimeters) of that point from the camera;

[0167] camera.fx represents the actual physical length (in millimeters) represented by each pixel in the horizontal direction of the image captured by camera 193; camera.fy represents the actual physical length (in millimeters) represented by each pixel in the vertical direction of the image captured by camera 193; camera.cx represents the horizontal pixel difference between the center point coordinates and the origin coordinates of the depth image; camera.cy represents the vertical pixel difference between the center point coordinates and the origin coordinates of the depth image.

[0168] It should be understood that different cameras have different intrinsic parameters. In some embodiments of this application, the camera intrinsic parameters of mobile phone 10 may be camera.scale = 1.0; camera.cx = 313.259979; camera.cy = 270.867126; camera.fx = 563.343384; camera.fy = 563.343384.

[0169] Then, using this mapping formula, the phone calculates the coordinates (px, py, pz) of each point in the point cloud corresponding to the depth image of the space where the object is located.

[0170] 2) Based on the ground region in the RGB image, the mobile phone 10 determines the ground region in the depth image corresponding to the RGB image, and converts the pixel data of the depth image corresponding to the ground region in the depth image into a ground point cloud.

[0171] Specifically, the mobile phone 10 uses the trained semantic segmentation model to identify the ground region in the RGB image of the space where the object to be tested is located; then, the mobile phone 10 determines the ground region in the depth image corresponding to the RGB image based on the ground region in the RGB image; finally, the mobile phone 10 determines the ground point cloud in the point cloud 1113b corresponding to the depth image based on the ground region in the depth image.

[0172] 3) Mobile phone 10 uses the above ground point cloud to fit one or more planes (i.e., sub-virtual planes).

[0173] It should be understood that ground point clouds correspond to ground regions in depth images. Therefore, if the ground is completely flat, the Z-coordinates of the corresponding ground point clouds should be the same. However, even in real-world scenes, the ground is not necessarily a perfectly flat plane. For example, the ground may be covered with carpets or other attachments, or it may have potholes, small bumps, etc. Therefore, an actual ground point cloud should be a collection of discrete points that can reflect the approximate shape of the ground; that is, the Z-coordinates of each point in the ground point cloud are not necessarily equal.

[0174] Therefore, the mobile phone 10 obtains the ground point cloud corresponding to the ground area in the space where the object to be tested is located through the above steps 1) to 2). In the process of fitting the ground of the AR scene according to the coordinates (px, py, pz) of each point in the ground point cloud (mainly Z coordinate, pz), it may fit one or more planes. Then, based on the distance between each plane and the camera 193 or the distance difference between each plane, further plane fitting is performed to form a larger plane.

[0175] For example, suppose there are 100 points in a ground point cloud. Among them, 20 points, such as P1, P2, P3...P20, have the same Z-coordinate of -8; 20 points, such as P21, P22, P23...P40, have the same Z-coordinate of -10; 20 points, such as P41, P42, P43...P60, have the same Z-coordinate of -9; 30 points, such as P61, P62, P63...P90, have the same Z-coordinate of -20; and the remaining 10 points, such as P91, P92, P93...P100, have different Z-coordinates.

[0176] Based on the Z-coordinates of the points mentioned above, the phone 10 will ultimately fit four planes. Specifically, the phone 10 will fit plane 1 to points P1, P2, P3...P20 with the same Z-coordinate of -8; plane 2 to points P21, P22, P23...P40 with the same Z-coordinate of -10; plane 3 to points P41, P42, P43...P60 with the same Z-coordinate of -9; and plane 4 to points P61, P62, P63...P90 with the same Z-coordinate of -20. Points P91, P92, P93...P100 with different Z-coordinates will be discarded. In other words, if all 100 points have the same Z-coordinate, the phone 10 can fit a single plane based on these 100 points, and use this plane as the ground in the AR scene.

[0177] 4) Mobile phone 10 calculates the distance from each of the above planes to camera 193.

[0178] Specifically, after fitting the aforementioned planes 1, 2, 3, and 4, the mobile phone 10 will calculate the distance from each of these planes to the camera 193 of the mobile phone 10. It can be understood that, as shown in the calculation formula in step 1), the coordinates of each point in the point cloud are relative to the camera 193. Therefore, the distance from the camera 193 to the plane fitted from each point in the ground point cloud is equal to the Z-coordinate of a point in each of the aforementioned planes.

[0179] Therefore, the distances of plane 1, plane 2, plane 3, and plane 4 from camera 193 are 8 cm, 10 cm, 9 cm, and 20 cm, respectively.

[0180] 5) The mobile phone 10 fits the planes in each plane that have a distance difference from the camera 193 within a preset distance until the largest plane is obtained. The mobile phone 10 uses the largest plane as the ground in the AR scene.

[0181] For example, the distance from plane 1 to camera 193 is 8 cm, from plane 2 to camera 193 is 10 cm, from plane 3 to camera 193 is 9 cm, and from plane 4 to camera 193 is 20 cm. Then, phone 10 will fit plane 1 and plane 3 into a new plane 5, and the distance from the new plane 5 to camera 193 will be the average of the distances from plane 1 and plane 3 to camera 193, which is 8.5 cm. Simultaneously, it will fit plane 2 and plane 3 into a new plane 6, and the distance from plane 6 to camera 193 will be the average of the distances from plane 2 and plane 3 to camera 193, which is 9.5 cm. Since the distance difference between plane 4 and planes 1, 2, and 3 to camera 193 is greater than 1 cm, plane 4 will be discarded during the ground fitting process.

[0182] Then, the phone 10 repeats the above process until the largest plane is obtained. For example, since the distance from plane 5 to camera 193 is 8.5 cm and the distance from plane 6 to camera 193 is 9.5 cm, the phone 10 continues to fit plane 5 to plane 6 to obtain a new plane 7, and the distance of plane 7 from camera 193 is 9 cm. Since plane 7 is the largest plane at this time, plane 7 is used as the ground in the AR scene.

[0183] It should be understood that when fitting the ground in an AR scene, the Z coordinates of each point in the point cloud can be used as the basis for fitting the ground. If fitting other planes in the AR scene, such as walls, the X coordinates of each point in the point cloud can be used as the basis for fitting the walls. If fitting other objects in the AR scene, the distance of each point in the point cloud from the camera 193 is taken into account for fitting. The specific plane fitting process is similar to the principle of steps 1) to 5) above, and will not be repeated here.

[0184] Optionally, the mobile phone 10 can first use a semantic segmentation model to identify the ground region in the RGB image, and then determine the ground region in the depth image corresponding to the RGB image based on the ground region in the RGB image. The mobile phone 10 then converts only the ground region in the depth image into a ground point cloud using the method in step 1), and fits the ground in the AR scene according to steps 2) to 5). Compared to the previous method where the mobile phone 10 converts the entire depth image into a corresponding point cloud, the mobile phone 10 only converts the ground region in the depth image into a ground point cloud, thus improving the efficiency of the mobile phone 10 in converting the depth image into a corresponding point cloud.

[0185] Furthermore, when the mobile phone 10 uses a semantic segmentation model to identify the ground in the aforementioned RGB image, the identified ground area may contain some edge points, such as points at the boundary between the ground and the wall or the boundary between the ground and the table. Moreover, these edge points are often discrete relative to other points in the concentrated ground point cloud. Correspondingly, these points also exist in the depth image corresponding to the RGB image. Therefore, when the mobile phone 10 fits the ground in the AR scene based on the ground point cloud, it needs to separately determine whether these points can be fitted into a plane with other points, affecting the efficiency of the mobile phone 10 in fitting the plane. Therefore, in one possible implementation of this application, the above-mentioned plane fitting process can also be:

[0186] First, the mobile phone 10 divides the ground point cloud into multiple spatial blocks, such as spatial block 1, spatial block 2, and spatial block 3, based on the coordinates of each point in the ground point cloud. The distance between the center of the spatial block and the mobile phone 10 can also be a preset distance, such as 5 cm, 6 cm, and 7 cm.

[0187] Then, the mobile phone 10 determines whether multiple spatial blocks contain points and whether the number of points meets a preset condition, such as 10. If a spatial block, such as spatial block 1, does not contain any points or contains points but the number of points is less than 10, then spatial block 1 is discarded, and only the points in spatial blocks 2 and 3 are used to fit a plane. Specifically, the planes in spatial block 2, such as planes 8, 9, and 10, can be fitted first based on the points in spatial block 2, and the planes in spatial block 3, such as planes 11 and 12, can be fitted based on the points in spatial block 3. Finally, the mobile phone 10 determines whether to fit the aforementioned planes into a larger plane based on the distances of planes 8, 9, 10, 11, and 12 from the camera 193 of the mobile phone 10. The process of fitting the plane within the spatial block is the same as the process of directly fitting a plane based on points in step 3) above, and will not be repeated here.

[0188] Additionally, because users are "mobile devices, searching for the ground" (e.g.) Figure 4As shown in Figure (a), during the process, the position of the mobile phone 10 will change as it moves closer to or further away from the person being tested. Synchronously with the change in the position of the mobile phone 10, the three-dimensional coordinates of the depth-sensing camera 193 will also change (assuming that the orientation of the depth-sensing camera 193 does not change). Accordingly, the display range of the depth image of the person being tested on the ground in the real scene, which is obtained by the mobile phone 10 through the depth-sensing camera 193, will also change accordingly.

[0189] For example, when the phone 10 is close to the person being tested, the depth image of the ground where the person is located in the real scene, obtained by the phone 10 through the depth-sensing camera 193, includes a smaller range and more details of the ground in the real scene. When the phone 10 is far away from the person being tested, the depth image of the ground where the person is located in the real scene, obtained by the phone 10 through the depth-sensing camera 193, includes a larger range and less details of the ground in the real scene.

[0190] It is understandable that in the above process, if the phone 10 only uses the depth image corresponding to a single pose of the depth sensing camera 193, it will be unable to fit the ground in the AR scene that is consistent with the ground in the real scene. For example, if the phone 10 uses the depth image of the ground corresponding to the pose of the depth sensing camera 193 near the subject to fit the ground in the AR scene, then the range of the fitted AR scene ground will be smaller than the range of the ground where the subject is located in the real scene.

[0191] Therefore, in some embodiments of this application, the mobile phone 10 will combine the depth images corresponding to the depth sensing camera 193 in different poses to fit the ground of the AR scene. In one possible implementation, the mobile phone 10 can obtain the acceleration and angular velocity 1112b of the mobile phone 10 at various moments during the process of the user slowly moving the mobile phone 10 through the IMU unit 180B, and then use the simultaneous localization and mapping (SLAM) algorithm to determine the current pose information 1114b of the camera 193, that is, the three-dimensional coordinates and orientation angle of the camera 193. It should be understood that since the camera 193 is mounted on the mobile phone 10, the acceleration and angular velocity of the mobile phone 10 during the process of being moved by the user can be regarded as the acceleration and angular velocity of the depth sensing camera 193 during the movement process. Optionally, the above-mentioned SLAM algorithm can be pre-developed as an SDK file and then pre-installed in the mobile phone 10. In this way, after the mobile phone 10 obtains the acceleration and angular velocity corresponding to the mobile phone 10 at each moment during the process of the user slowly moving the mobile phone 10 through the IMU unit 180B, the IMU unit 180B inputs the obtained acceleration and angular velocity corresponding to the mobile phone 10 at each moment to the SLAM algorithm software, and then obtains the three-dimensional coordinates and orientation angle values ​​of the camera 193 through relevant calculations.

[0192] Meanwhile, when users are "slowly moving their devices, searching for the ground" (reference) Figure 4 During the process shown in Figure (a), the depth-sensing camera 193 of the mobile phone 10 continuously acquires RGB-D images of the space where the person being tested is located. Based on the depth images of the space where the person being tested is located acquired by the depth-sensing camera 193, the mobile phone 10 fits multiple planes in the AR scene according to the methods in steps 1) to 5) above. At the same time, the mobile phone 10 uses the semantic segmentation model trained above to perform semantic segmentation 1120 on the RGB image corresponding to the aforementioned depth image to determine the type of each plane in the RGB image, that is, the semantic label 1123b.

[0193] As mentioned earlier, there is a one-to-one correspondence between the pixels of the RGB image and the depth image in an RGB-D image, and a one-to-one correspondence between each point in the depth image and each point in the point cloud obtained from the depth image. Therefore, there is also a one-to-one correspondence between each point in the RGB image and each point in the point cloud. Thus, the mobile phone 10 can map the type of each plane in the RGB image to the type of plane in the AR scene fitted based on the point cloud. That is, the mobile phone 10 can add semantic tags 1123b representing the plane type to multiple planes in the fitted AR scene based on the type of plane in the RGB image, and then generate a plane 1116b with semantic tags 1123b.

[0194] For example, the mobile phone 10 uses a trained semantic segmentation model to determine the ground, walls, and other planes in the RGB image of the space where the test subject is located in the real scene. Then, the mobile phone 10 marks the "ground" onto the plane fitted in the AR scene based on the depth image corresponding to the ground in the real scene, and marks the "wall" onto the plane fitted in the AR scene based on the depth image corresponding to the wall in the real scene.

[0195] As shown above, the pose of the depth-sensing camera 193 changes continuously as the phone 10 is moved by the user. For example, when the phone 10 moves closer to or further away from the person being tested, the area of ​​the ground included in the RGB image acquired by the depth-sensing camera 193 differs, and correspondingly, the point cloud of the ground in the depth image corresponding to the RGB image also differs. Therefore, the phone 10 calculates the plane confidence of multiple planes fitted based on the point clouds corresponding to the depth images acquired by the depth-sensing camera 193 under different poses. The plane confidence is the ratio between the number of points contained in the fitted plane and the number of points in the point cloud corresponding to that plane in the depth image. The calculation process of the plane confidence will be described below.

[0196] Then, the phone uses the plane type in the RGB image and the plane confidence of the fitted plane as labels for multiple planes in the fitted AR scene, denoted as Tag{type, confidence}.

[0197] Finally, the mobile phone 10 fits the ground in the AR scene based on the RGB-D images of the space where the object under test is located, obtained by the camera 193 in various poses, and fits the corresponding Tag content of each plane in the AR scene, so that the ground in the AR scene is closer to the ground in the real scene.

[0198] For example, suppose the user slowly moves the phone 10 to find the ground, acquiring multiple RGB-D images in different poses. Then, using the methods described in steps 1) to 5), the phone 10 fits 10 planes of type 'ground' with a confidence level of 90%, and 5 planes of type 'ground' with a confidence level of 30%, based on the RGB-D images in different poses. When fitting the ground in an AR scene using the 10 planes of type 'ground' with a confidence level of 90% and the 5 planes of type 'ground' with a confidence level of 30%, the phone 10 can select the 10 planes of type 'ground' with higher confidence for plane fitting. The 5 planes of type 'ground' with a confidence level of only 30% can be discarded or used as plane references with reduced weights for plane fitting. It can be understood that the more RGB-D images of the ground in the real scene acquired by the camera 193 in different poses, the more planes of type 'ground' the phone 10 can fit, and the closer the ground in the AR scene fitted by the phone 10 is to the ground in the real scene.

[0199] Optionally, the planes with Tag{type, confidence} mentioned above can be used as a dataset for training the semantic segmentation model again to further optimize the semantic segmentation model and make the plane types identified by the semantic segmentation model more accurate.

[0200] For example, if mobile phone 10 uses the method described above to fit 5 planes of type ground with a confidence level of 90%, and 5 planes of type wall with a confidence level of 80%, mobile phone 10 can use the 5 planes of type ground with a confidence level of 90% as the ground in the dataset (or use the 5 planes of type wall with a confidence level of 80% as the walls in the dataset), and input these planes as the target data in the semantic segmentation training into the semantic segmentation model. The semantic segmentation model is then trained until the confidence level of the plane output by the semantic segmentation model is the same as or similar to that of the planes of type ground with a confidence level of 90% (or the confidence level of the planes of type wall with a confidence level of 80%). At this point, the semantic segmentation model is considered to have completed training. The semantic segmentation model is continuously optimized in this way.

[0201] Corresponding to the plane confidence level mentioned above, the calculation method for the plane confidence level is briefly introduced below:

[0202] During the aforementioned plane fitting process, the camera 193 of the mobile phone 10 continuously scans the fitted plane to obtain the distance between the fitted plane and the camera 193. Then, based on the distance between the fitted plane and the camera 193, it determines whether to continue fitting a portion of the fitted plane to form a larger plane. For example, a space may include three fitted planes L, M, and N. The mobile phone 10 obtains through the camera 193 that the distances of the three smaller planes L, M, and N from the camera 193 are 10 cm, 9 cm, and 30 cm, respectively. The mobile phone 10 then determines whether the three smaller planes can be fitted into a single plane based on whether the difference between the distances between each smaller plane and the camera 193 is greater than a preset distance. Taking a preset distance of 1 cm as an example, the distances between the aforementioned L plane and M plane and camera 193 are 10 cm and 9 cm respectively. The difference between these distances is equal to the preset distance of 1 cm, so L plane and M plane can be fitted into one plane. However, the distance difference between N plane and the other two planes and camera 193 is greater than the preset distance of 1 cm, so N plane will be discarded during the plane fitting process. Then, phone 10 will calculate the average distance between L plane and N plane and camera 193, for example, (10+9) / 2 = 9.5 cm. Therefore, phone 10 will combine the point clouds of L plane and N plane in the space 9.5 cm away from camera 193 to fit a new plane O. Assuming that L plane contains 100 point clouds, M plane contains 200 point clouds, and N plane contains 200 point clouds, then the plane confidence of the new plane O is: (100+200) / (100+200+200) = 60%. It is understood that the plane confidence score can also be calculated by calculating the ratio of the number of point clouds used to fit the plane to the total number of point clouds in the space. This application does not limit the calculation method of the plane confidence score.

[0203] It should be understood that the size measurement method of this application can not only measure the size of the target perpendicular to the ground, such as a person's height, but also measure the size of the target perpendicular to a wall. In this case, the mobile phone 10 will identify the wall from the acquired RGB image, and then determine the depth data of the wall in the depth image corresponding to the RGB image based on the position information of the wall. The depth data of the wall will be converted into a corresponding point cloud. The wall in the AR scene will be fitted based on the point cloud, and the wall will be used as the measurement reference plane for measuring the target. The size of the target perpendicular to the wall will be measured. The specific measurement principle is the same as that for measuring a person's height. For details, please refer to the relevant description of measuring a person's height above, which will not be repeated here.

[0204] In another possible implementation of this application, the way the mobile phone 10 identifies the vertex P1 of the human head in step 1103 can be that the mobile phone 10 acquires an image of the human through the camera 193, then uses a two-dimensional image recognition method to identify the outline of the human head, and takes the highest point of the head outline as the vertex of the human head, and calculates the distance from the vertex of the head to the aforementioned measurement reference plane.

[0205] Figure 13 This is a schematic diagram illustrating an example of using an image recognition method to identify the outline of a human head, as provided in an embodiment of this application.

[0206] like Figure 13 As shown, the mobile phone 10 acquires a two-dimensional image A including a human head through the camera 193, then uses an image recognition method based on a convolutional neural network model to determine the contour L of the human face, and takes the highest point P1 of the contour as the vertex of the human head. Then, the distance from P1 to the aforementioned measurement reference plane is calculated, and finally, this distance is displayed on the mobile phone 10 as the human's height.

[0207] Since the above method of identifying the vertex P1 of a person's head is based on a two-dimensional image, the coordinates of the vertex P1 at this time belong to the coordinates in the image coordinate system, that is, two-dimensional coordinates, with only two values, XY horizontal and vertical coordinates, and no Z-axis coordinate that can represent the depth data of the vertex P1.

[0208] The measurement reference plane α is a plane in the world coordinate system. Therefore, when calculating the distance from the head vertex P1 to α, it is necessary to convert the head vertex P1 to coordinates in the world coordinate system (Xp1, Yp1, Zp1).

[0209] It is understandable that the image coordinate system is a two-dimensional coordinate system, while the world coordinate system is a three-dimensional coordinate system. The two-dimensional coordinate system lacks depth data compared to the three-dimensional coordinate system, meaning that points in the two-dimensional coordinate system do not have corresponding depths. Therefore, when determining the three-dimensional coordinates of a point in the two-dimensional coordinate system, the distance from that point to camera 193 is determined based on the coordinates of points in multiple two-dimensional images. Then, a series of rotations and translations are performed to obtain the three-dimensional coordinates of that point. During this process, since the poses of the multiple two-dimensional images acquired by camera 193 may be different, and the two-dimensional images selected to determine the three-dimensional coordinates of one point and another point may also be different, there will inevitably be errors in the process of converting the two-dimensional image coordinate system to the three-dimensional image coordinate system. That is, there will be unavoidable errors in converting the coordinates of the head vertex P1 from the above image coordinate system to the world coordinate system, which in turn leads to inaccurate human height measurements.

[0210] Therefore, in order to further improve the accuracy of human height measurement, in another possible implementation of this application, the mobile phone 10 can determine the vertex of the human head by acquiring a human facial depth image, then acquiring the first facial feature point based on the facial depth image, and then calculating the first facial feature point or determining the distance of the vertex of the human head from the measurement reference plane based on the first facial feature point. Figure 14(a) is a flowchart of a method for determining the vertex of a human head and calculating human height using three-dimensional face recognition technology provided by some embodiments of this application.

[0211] As shown in Figure 14(a), method 1400 includes:

[0212] 1401: Mobile phone 10 acquires a depth image of a person's face.

[0213] In some embodiments of this application, the mobile phone 10 acquires a person's facial depth image through the depth-sensing camera 193.

[0214] 1402: Based on the above facial depth image, mobile phone 10 determines the first feature point of a person's face.

[0215] In one possible implementation of this application, the mobile phone 10 uses three-dimensional face recognition technology to identify the facial depth image in step 1401 and determine the first feature point of the human face.

[0216] The 3D face recognition technology primarily utilizes deep learning algorithms such as face detection, feature point calibration, 3D greedy mapping surface reconstruction, shortest path, isometric mapping, and matrix K-order moments to extract 3D face features and then recognizes the face based on these extracted features. In this application, the mobile phone 10 can identify feature points in a 3D face, i.e., the first facial feature point, through 3D face recognition technology. Optionally, the first facial feature point can be a prominent feature such as the tip of the nose, forehead, cheeks, brow bone, or bridge of the nose; or a distinct corner such as the corner of the eye, wing of the nose, or corner of the mouth. This application does not limit the specific form of the first facial feature point.

[0217] Optionally, the aforementioned 3D face recognition technology can be integrated into the face recognition SDK of mobile phone 10. When mobile phone 10 needs to recognize a face, it will call the face recognition SDK. The face SDK can be integrated into the face AR recognition module of mobile phone 10.

[0218] 1403: Mobile phone 10 determines the top of a person's head based on the first feature point of the face, and determines the distance from the top of the person's head to the ground in the AR scene, and uses the distance from the top of the person's head to the ground in the AR scene as the person's height.

[0219] Since the tip of the nose is the most prominent feature point on a human face, it is the easiest to identify in a facial depth image. Therefore, taking the tip of the nose as the first feature point of the subject's face identified by the mobile phone 10 as an example, in one possible implementation, the mobile phone 10 obtains the coordinates of the subject's nose tip, and then, based on the proportions of the human face, further determines the distance from the top of the subject's head to the tip of the nose according to the nose tip coordinates. Then, the sum of the distance from the top of the subject's head to the tip of the nose and the distance from the tip of the nose to the ground is taken as the height of the subject.

[0220] Figure 14(b) is a flowchart of a face model generated using three-dimensional face recognition technology combined with a neural network model, provided in some embodiments of this application.

[0221] As shown in Figure 14(b), it includes a facial depth image acquisition module 1401b, a 3D face detection module 1402b, a marker generation module 1403b, and a face mesh generation module 1404b.

[0222] The facial depth image acquisition module 1401b is used to implement step 1401 above. The 3D face detection module 1402b and the marker generation module 1403b work together to implement steps 1402-1403 above. Specifically, the 3D face detection module 1402b identifies the facial depth image, the marker generation module 1403b marks all facial feature points, and finally, the face mesh generation module 1404b generates a face mesh with the marked facial feature points. It should be understood that the specific implementation process of each module corresponds to each step in method 1400, and can be referred to the description of method 1400 for details, which will not be repeated here.

[0223] After generating the face mesh using the process shown in Figure 14(b), the mobile phone 10 will determine the position of the head vertex based on the facial proportions of the face in the face mesh. Specifically, Figure 15 This is a schematic diagram illustrating an example of determining the position of the top of a human head, provided by some embodiments of this application.

[0224] like Figure 15 As shown, the mobile phone 10 can determine the coordinates of the nose tip 29 in the world coordinate system based on the depth image of the human face, and then determine the coordinates of the head vertex 63 (i.e., P1) of the person being tested based on the proportions of the human face. For example, assuming the coordinates of the nose tip 29 are (x1, y1, z1) and the coordinates of the point 62 in the middle of the eyebrows are (x2, y2, z2), in the proportions of the human face, the distance from the head vertex 63 to the point 62 in the middle of the eyebrows is equal to the distance from the nose tip 29 to the point 62 in the middle of the eyebrows, and also equal to the distance from the nose tip 29 to the chin 54.

[0225] Therefore, a person's height is equal to: the distance from the tip of the nose 29 to the ground + the distance from the tip of the nose 29 to the top of the head 63, which is also equal to the distance from the tip of the nose 29 to the ground + twice the distance from the tip of the nose 29 to the chin 54, or the distance from the tip of the nose 29 to the ground + twice the distance to the point 62 between the eyebrows.

[0226] After determining a person's height, the phone will calculate the height according to... Figures 4 to 9 The method for displaying average height is shown in the UI of the phone. Please refer to [link / reference]. Figures 4 to 9 The description in the text will not be repeated here.

[0227] Figure 16This is a schematic diagram of the software structure of a mobile phone 10 according to an embodiment of this application. Taking the Android system as the operating system of the mobile phone 10 as an example, in some embodiments, the Android system is divided into four layers, namely the application layer 1601, the application framework layer (FWK) 1602, the hardware abstraction layer (HAL) 1603, the kernel layer 1604, etc.

[0228] like Figure 16 As shown, the aforementioned application layer can be a series of application packages, which may include applications such as camera 1601a, calendar 1601b, SMS 1601c, AR measurement 1601d, and navigation 1601e. These will not be listed individually in the embodiments of this application.

[0229] The application framework layer 1602 provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer 1602 includes predefined function interfaces, such as event listener functions for receiving events sent by the application framework layer.

[0230] In some embodiments of this application, the event listener function of the application framework layer 1602 will receive the user's opening of the "AR Measurement" APP and the touch operation within the "AR Measurement" APP, so as to call the corresponding hardware service according to the user's specific touch operation.

[0231] HAL 1603 is an interface layer located above the hardware circuitry in the operating system kernel layer 1604 and physical layer 1605. Its purpose is to abstract the hardware. HAL 1603 shields the implementation details of the hardware in the phone 10 from the lower layers and provides an abstract interface for the application layer 1601 to call the hardware services in the phone 10.

[0232] In one possible implementation of this application, the event listener function of the application framework layer 1602 will receive the user's opening of the "AR Measurement" APP and the touch operation within the "AR Measurement" APP. In response to the user's touch operation, the application framework layer 1602 will, as follows: Figure 16 The ServiceManager 1602a interface shown calls services from specific hardware in the physical layer 1605. For example, when a user initiates AR measurement 1601d in the application layer 1601 to measure the height of the person being measured, and the mobile phone 10 is moved to find the ground, it responds to the user's selection of the "height measurement" mode (such as...). Figure 4 Figure (a) and Figure 4As shown in Figure (b), the application "AR Measurement" will call the camera driver 1604a and sensor driver 1604b in kernel layer 1604 (e.g., hw_get_module 1603a) via the ServiceManager.getService() function in the ServiceManager 1602a interface to drive the camera 193 (e.g., ...). Figure 10 (as shown in Figure (a)) acquires an RGB-D image of the ground within the space where the subject is located, and simultaneously uses the IMU unit 180B of the mobile phone 10 (as shown in Figure (a)) to obtain the image. Figure 10 (as shown in Figure (a)) The pose information of camera 193 is obtained so that mobile phone 10 can combine the depth image of the ground in the space where the person being tested is located with the pose information of camera 193 to perform plane fitting in AR scene.

[0233] Kernel layer 1604 is the layer between hardware and software. Kernel layer 1604 includes at least camera driver 1604a and sensor driver 1604b, and may also include display driver, microphone driver, etc.

[0234] It should be understood that the software architecture layering of the mobile phone 10 described above is merely exemplary and does not constitute a limitation of this application. In some other implementations, the software architecture layering of the mobile phone 10 may be divided into more or fewer layers according to the layering principle. For example, the software architecture layering of the mobile phone 10 may also include a system layer and a physical layer, or the software architecture layering of the mobile phone 10 may not include HAL 1603. This application does not impose any limitations on this.

[0235] This application implements all or part of the processes in the methods of the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0236] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0237] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0238] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In the foregoing description, specific details such as particular system architectures and techniques have been set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail that could obscure the description of this application.

[0241] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0242] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0243] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0244] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0245] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0246] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for measuring the size of a target, applied to electronic devices, characterized in that, include: During the movement of the electronic device, two-dimensional and three-dimensional information images of the target in the real scene are acquired by the camera, and the acceleration and angular velocity data of the camera at various moments are acquired by the inertial measurement unit. The pixels in the two-dimensional and three-dimensional information images have a one-to-one correspondence in position. Based on the acceleration and angular velocity data, the pose information of the camera at each moment is determined; An AR scene is created based on the three-dimensional information image; The ground in the two-dimensional information image is identified using a semantic segmentation model; Based on the location information of the ground in the two-dimensional information image, obtain the three-dimensional pixel information of the pixels corresponding to the location information in the three-dimensional information image; The acquired 3D pixel information is converted into a point cloud, and the ground in the AR scene is fitted based on the pose information of the camera at each moment and the converted point cloud. The dimensions of the target to be measured are taken using the ground in the AR scene as a reference.

2. The method according to claim 1, characterized in that, The method of acquiring two-dimensional and three-dimensional information images of the target in a real scene based on camera acquisition includes: The depth-sensing camera of the electronic device simultaneously acquires two-dimensional and three-dimensional information images of the ground where the target is located.

3. The method according to claim 1 or 2, characterized in that, The two-dimensional information image represents the two-dimensional features of the target under test, and the two-dimensional features include one or more of color features, grayscale features, and texture features. The three-dimensional information image represents the three-dimensional features of the target under test, and the three-dimensional features include the spatial depth value of the target under test.

4. The method according to claim 1, characterized in that, The two-dimensional information image includes a color space image, which includes an RGB image or a YUV image, and the three-dimensional information image includes a depth image.

5. The method according to claim 1, characterized in that, The semantic segmentation model is a fully convolutional neural network (FCNs) model.

6. The method according to claim 1, characterized in that, The step of converting the acquired 3D pixel information into a point cloud and generating a virtual plane corresponding to the ground in the AR scene based on the converted point cloud includes: Based on the converted point cloud, multiple sub-virtual planes corresponding to the ground in the AR scene are generated; The plane confidence score of each of the plurality of sub-virtual planes is determined, and the virtual plane is generated based on a portion of the sub-virtual planes, wherein the plane confidence score represents the ratio between the number of points in the point cloud of each sub-virtual plane and the number of points in the point cloud corresponding to the ground.

7. The method according to claim 1, characterized in that, The target to be tested includes a person, and the method further includes: Obtain a human head image; Determine the vertex of the human head from the head image; The distance between the top of the head and the ground in the AR scene is taken as the height of the person.

8. The method according to claim 7, characterized in that, The head image is a three-dimensional information image of the head, and Determining the vertex of a person's head from the head image includes: Facial feature points are identified from a three-dimensional image of a person's head using a three-dimensional face recognition method. The head vertex is determined based on the facial feature points.

9. A computer-readable medium, characterized in that, The computer-readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method for measuring the size of the target as described in any one of claims 1-8.

10. An electronic device, characterized in that, include: Camera, Inertial Measurement Unit (IMU); One or more processors; One or more memory units; A module with multiple applications installed; The memory stores one or more programs, the one or more programs including instructions that, when executed by the electronic device, cause the electronic device to perform a method for measuring the size of the target as claimed in any one of claims 1 to 8.

11. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the processor to execute the method for measuring the size of the target as described in any one of claims 1 to 8.

Images