Depth detection methods, apparatus, electronic devices and computer-readable storage media

Through image segmentation and key point detection, combined with two-dimensional and three-dimensional information, multi-person depth detection is realized without a three-dimensional depth camera, solving the problem of insufficient depth estimation in the prior art, and is suitable for a variety of application scenarios.

CN112465890BActive Publication Date: 2024-11-08NANJING MINGZHISUI NETWORK TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011335257.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-24
Publication Date
2024-11-08
Estimated Expiration
2040-11-24

AI Technical Summary

Technical Problem

In the absence of special hardware equipment such as three-dimensional depth cameras, how to achieve multi-person depth detection in images is an urgent problem to be solved. The existing technology has insufficient depth estimation accuracy and information volume, and has increased time and computing resources. Consumption limits the application range.

Method used

By acquiring images collected by the image acquisition device, human image segmentation and key point detection are performed, and the depth detection results of multiple human bodies are determined without relying on a three-dimensional depth camera, using two-dimensional key points and masks Image matching and optimization processing realize interpolation and smoothing of depth information.

Benefits of technology

Without relying on a three-dimensional depth camera, depth detection of multiple human bodies in the image is realized, which reduces time and computing resources consumption, improves the accuracy and application range of depth information, and is suitable for scenes such as AR interaction and virtual photography.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112465890B_ABST
    Figure CN112465890B_ABST
Patent Text Reader

Abstract

The present disclosure provides a depth detection method, device, electronic device and computer-readable storage medium, the method comprising: acquiring at least one frame of image captured by an image acquisition device, the at least one frame of image comprising a current frame of image; performing human body image segmentation on the current frame of image to obtain mask images of multiple human bodies; and performing human body key point detection on the at least one frame of image to obtain two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame of image; determining depth detection results of the multiple human bodies in the current frame of image based on the two-dimensional key point information and three-dimensional key point information of the multiple human bodies in the current frame of image and the mask images of the multiple human bodies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to computer vision processing technology, and in particular to a depth detection method, device, electronic device and computer-readable storage medium. Background Art

[0002] Among the related technologies, multi-person depth detection technology of images has important applications in augmented reality (AR) interaction, virtual photography and other applications. In scenarios where special hardware equipment such as three-dimensional depth cameras are lacking, how to achieve multi-person depth detection of images is a technical problem that needs to be solved urgently. Summary of the invention

[0003] Embodiments of the present disclosure provide a depth detection method, device, electronic device, and computer-readable storage medium.

[0004] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0005] The present disclosure provides a depth detection method, the method comprising:

[0006] Acquire at least one frame of image captured by an image acquisition device, wherein the at least one frame of image includes a current frame of image;

[0007] Performing human body image segmentation on the current frame image to obtain multiple human body mask images; and performing human body key point detection on the at least one frame image to obtain two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame image;

[0008] Determine depth detection results of the multiple human bodies in the current frame image according to the two-dimensional key point information and the three-dimensional key point information of the multiple human bodies in the current frame image and the mask images of the multiple human bodies.

[0009] In some embodiments of the present disclosure, determining depth detection results of the multiple human bodies in the current frame image according to the two-dimensional key point information and the three-dimensional key point information of the multiple human bodies in the current frame image, and the mask images of the multiple human bodies, includes:

[0010] By matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, the two-dimensional key point information respectively belonging to each human body is obtained;

[0011] The depth detection result of each human body in the current frame image is determined according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body.

[0012] In some embodiments of the present disclosure, the two-dimensional key point information of each human body in the multiple human bodies is matched with the mask image of each human body in the mask images of the multiple human bodies to obtain the two-dimensional key point information belonging to each human body, including:

[0013] Among the two-dimensional key point information of the plurality of human bodies, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

[0014] In some embodiments of the present disclosure, determining the depth detection result of each human body in the current frame image according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body includes:

[0015] Determine the coordinate information of the three-dimensional key point information corresponding to the two-dimensional key point information of each human body; determine the depth information of the two-dimensional key point of each human body according to the coordinate information of the three-dimensional key point; interpolate the depth information of the two-dimensional key point of each human body to obtain the depth information of the first pixel point in the mask image of each human body; wherein the first pixel point represents any pixel point in the mask image of each human body except the pixel point overlapping with the position of the two-dimensional key point.

[0016] In some embodiments of the present disclosure, the two-dimensional key point information of each human body in the multiple human bodies is matched with the mask image of each human body in the mask images of the multiple human bodies to obtain the two-dimensional key point information belonging to each human body, including:

[0017] Optimizing the two-dimensional key point information of the plurality of human bodies in the current frame image to obtain the optimized two-dimensional key point information of the plurality of human bodies;

[0018] Among the two-dimensional key point information of the plurality of human bodies after the optimization process, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

[0019] In some embodiments of the present disclosure, the optimizing process of the two-dimensional key point information of the multiple human bodies in the current frame image to obtain the optimized two-dimensional key point information of the multiple human bodies includes:

[0020] In the case where the at least one frame image also includes a historical frame image, the two-dimensional key point information of multiple human bodies in the current frame image and the two-dimensional key point information of multiple human bodies in the historical frame image are processed to obtain the two-dimensional key point information of multiple human bodies after optimization.

[0021] In some embodiments of the present disclosure, the method further comprises:

[0022] Determine, according to the depth detection results of the multiple human bodies in the current frame image, a positional relationship between the multiple human bodies and at least one target object in the AR scene;

[0023] Based on the positional relationship, determining a combined presentation mode of the plurality of human bodies and the at least one target object;

[0024] Based on the combined presentation method, an AR effect of superimposing the multiple human bodies and the at least one target object is displayed.

[0025] In some embodiments of the present disclosure, at least one frame of image captured by the image capture device is a red, green, and blue (RGB) image.

[0026] In some embodiments of the present disclosure, the two-dimensional key point information is two-dimensional key points representing a human skeleton, and the three-dimensional key point information is three-dimensional key points representing a human skeleton.

[0027] The present disclosure also provides a depth detection device, the device comprising:

[0028] An acquisition module, used to acquire at least one frame of image acquired by an image acquisition device, wherein the at least one frame of image includes a current frame of image;

[0029] A processing module, used to segment the human body image of the current frame image to obtain a plurality of human body mask images; and detect human body key points of the at least one frame image to obtain two-dimensional key point information and three-dimensional key point information of a plurality of human bodies in the current frame image;

[0030] The detection module is used to determine the depth detection results of the multiple human bodies in the current frame image according to the two-dimensional key point information and the three-dimensional key point information of the multiple human bodies in the current frame image and the mask images of the multiple human bodies.

[0031] In some embodiments of the present disclosure, the detection module is used to determine the depth detection results of the multiple human bodies in the current frame image according to the two-dimensional key point information and the three-dimensional key point information of the multiple human bodies in the current frame image, and the mask images of the multiple human bodies, including:

[0032] By matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, the two-dimensional key point information respectively belonging to each human body is obtained;

[0033] The depth detection result of each human body in the current frame image is determined according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body.

[0034] In some embodiments of the present disclosure, the detection module is used to obtain two-dimensional key point information belonging to each human body by matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, including:

[0035] Among the two-dimensional key point information of the plurality of human bodies, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

[0036] In some embodiments of the present disclosure, the detection module is used to determine the depth detection result of each human body in the current frame image according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body, including:

[0037] Determine the coordinate information of the three-dimensional key point information corresponding to the two-dimensional key point information of each human body; determine the depth information of the two-dimensional key point of each human body according to the coordinate information of the three-dimensional key point; interpolate the depth information of the two-dimensional key point of each human body to obtain the depth information of the first pixel point in the mask image of each human body; wherein the first pixel point represents any pixel point in the mask image of each human body except the pixel point overlapping with the position of the two-dimensional key point.

[0038] In some embodiments of the present disclosure, the detection module is used to obtain two-dimensional key point information belonging to each human body by matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, including:

[0039] Optimizing the two-dimensional key point information of the plurality of human bodies in the current frame image to obtain the optimized two-dimensional key point information of the plurality of human bodies;

[0040] Among the two-dimensional key point information of the plurality of human bodies after the optimization process, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

[0041] In some embodiments of the present disclosure, the detection module is used to optimize the two-dimensional key point information of multiple human bodies in the current frame image to obtain the optimized two-dimensional key point information of multiple human bodies, including:

[0042] In the case where the at least one frame image also includes a historical frame image, the two-dimensional key point information of multiple human bodies in the current frame image and the two-dimensional key point information of multiple human bodies in the historical frame image are processed to obtain the two-dimensional key point information of multiple human bodies after optimization.

[0043] In some embodiments of the present disclosure, the processing module is further used to:

[0044] Determine, according to the depth detection results of the multiple human bodies in the current frame image, a positional relationship between the multiple human bodies and at least one target object in the AR scene;

[0045] Based on the positional relationship, determining a combined presentation mode of the plurality of human bodies and the at least one target object;

[0046] Based on the combined presentation method, an AR effect of superimposing the multiple human bodies and the at least one target object is displayed.

[0047] In some embodiments of the present disclosure, at least one frame of image captured by the image acquisition device is an RGB image.

[0048] In some embodiments of the present disclosure, the two-dimensional key point information is two-dimensional key points representing a human skeleton, and the three-dimensional key point information is three-dimensional key points representing a human skeleton.

[0049] The present disclosure also provides an electronic device, the electronic device comprising:

[0050] A memory for storing executable instructions;

[0051] The processor is used to execute the executable instructions stored in the memory to implement any one of the above-mentioned depth detection methods.

[0052] The embodiment of the present disclosure also provides a computer-readable storage medium storing executable instructions for implementing any of the above-mentioned depth detection methods when executed by a processor.

[0053] The embodiments of the present disclosure have the following beneficial effects:

[0054] The embodiments of the present disclosure can combine human body mask images of multiple human bodies as well as two-dimensional key points and three-dimensional key information of multiple human bodies to determine depth detection results of multiple human bodies, without the need to obtain depth information of the human bodies in the image through special hardware devices such as three-dimensional depth cameras. Therefore, the embodiments of the present disclosure can realize depth detection of multiple human bodies in the image without relying on special hardware devices such as three-dimensional depth cameras, and can be applied to scenarios such as AR interaction and virtual photography. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1is a schematic diagram of a connection between a terminal and a server provided in an embodiment of the present disclosure;

[0056] Figure 2 is an optional flowchart of the depth detection method provided by the embodiment of the present disclosure;

[0057] Figure 3 is a schematic diagram of two-dimensional key points of a human skeleton provided by an embodiment of the present disclosure;

[0058] Figure 4 is another optional flowchart of the depth detection method provided by the embodiment of the present disclosure;

[0059] Figure 5 is a schematic diagram of a point cloud provided by an embodiment of the present disclosure;

[0060] Figure 6 is a schematic diagram of an optional structure of a depth detection device provided in an embodiment of the present disclosure;

[0061] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are only used to explain the present disclosure and are not intended to limit the present disclosure. In addition, the embodiments provided below are partial embodiments for implementing the present disclosure, rather than providing all embodiments for implementing the present disclosure. In the absence of conflict, the technical solutions recorded in the embodiments of the present disclosure can be implemented in any combination.

[0063] It should be noted that, in the embodiments of the present disclosure, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements (such as steps in a method or units in a device, for example, a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.) in the method or apparatus including the element.

[0064] For example, the depth detection method provided in the embodiment of the present disclosure includes a series of steps, but the depth detection method provided in the embodiment of the present disclosure is not limited to the recorded steps. Similarly, the depth detection device provided in the embodiment of the present disclosure includes a series of modules, but the device provided in the embodiment of the present disclosure is not limited to including the modules explicitly recorded, and may also include modules that need to be set up to obtain relevant information or perform processing based on information.

[0065] The term "and / or" herein is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or C can represent the three situations of: A exists alone, A and C exist at the same time, and C exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, C, and D can represent including any one or more elements selected from the set consisting of A, C, and D.

[0066] In the related technology, special hardware such as three-dimensional depth cameras can be used to realize the depth detection of the human body in the image. The three-dimensional depth camera here can be a camera with a binocular camera and uses binocular vision technology to obtain depth information; however, the use of these special hardware will increase the application cost and limit the application scenarios to a certain extent.

[0067] When estimating the depth of the human body based on images taken by a monocular camera, the requirements for the accuracy of the depth estimation and the amount of information provided are relatively low; in some embodiments, when estimating the depth of the human body based on images taken by a monocular camera, only the relative depth between each pixel of the human body can be estimated, and the depth between the pixel of the human body and the camera cannot be estimated, which limits the scope of application to a certain extent; in some embodiments, only a single depth can be estimated for each pixel of the human body, and the estimated depth information is less; in other embodiments, depth information estimation can be achieved based on an image matching algorithm for continuous frames, but this solution increases the consumption of time resources and computing resources and is not suitable for low-power real-time application scenarios.

[0068] In response to the above technical problems, the embodiments of the present disclosure provide a depth detection method, device, electronic device and computer-readable storage medium, which can realize single-person depth detection in an image without relying on special hardware devices such as a three-dimensional depth camera; the depth detection method provided by the embodiments of the present disclosure can be applied in electronic devices. The exemplary application of the electronic device provided by the embodiments of the present disclosure is described below.

[0069] In some embodiments, the electronic device provided by the embodiments of the present disclosure can be implemented as AR glasses, laptops, tablet computers, desktop computers, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and other terminals with image acquisition devices. The image acquisition device can be a monocular camera or other device. Exemplarily, the terminal can be a mobile phone with a camera, where the mobile phone can be moved by hand.

[0070] Exemplarily, after receiving the image captured by the image acquisition device, the terminal can perform depth detection on the image captured by the image acquisition device according to the depth detection method of the embodiment of the present disclosure to obtain depth detection results of multiple human bodies in the image.

[0071] In some embodiments, the electronic device provided by the embodiments of the present disclosure may also be implemented as a server that forms a communication connection with the above-mentioned terminal. Figure 1 is a schematic diagram of the connection between the terminal and the server in the embodiment of the present disclosure. Figure 1 As shown, the terminal 100 is connected to the server 102 via the network 101. The network 101 may be a wide area network or a local area network, or a combination of the two.

[0072] In some embodiments, the server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0073] The terminal 100 is used to capture images at the current moving position through an image capture device; the captured images can be sent to the server 102; after receiving the images, the server 102 can perform depth detection on the received images according to the depth detection method of the embodiment of the present disclosure to obtain depth detection results of multiple human bodies in the image.

[0074] The depth detection method of the embodiment of the present disclosure is described below in combination with the above-mentioned contents.

[0075] Figure 2 FIG. 1 is an optional flow chart of the depth detection method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process may include:

[0076] Step 201: Acquire at least one frame of image captured by an image acquisition device, wherein the at least one frame of image includes a current frame of image.

[0077] In the embodiment of the present disclosure, the image acquisition device can acquire images and can send at least one frame of images including the current frame of images to a processor of the electronic device.

[0078] In some embodiments, at least one frame image includes a current frame image (a frame image captured at the current moment); in some embodiments, at least one frame image includes not only a current frame image, but also a historical frame image, where the historical frame image represents one or more frames of historical images captured by the image acquisition device.

[0079] In some embodiments, when at least one frame of image is a multi-frame image, the at least one frame of image may be a continuous frame image continuously captured by an image acquisition device, or may be a discontinuous multi-frame image, which is not limited in this embodiment of the present disclosure.

[0080] Step 202: segment the human body image of the current frame image to obtain mask images of multiple human bodies; and detect human body key points on at least one frame image to obtain two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame image.

[0081] In the disclosed embodiment, the human body image may be segmented for the current frame image according to a pre-trained image segmentation model to obtain human body mask images of multiple human bodies.

[0082] In the embodiments of the present disclosure, the image segmentation model may be a model related to the attributes of the human body image. In some embodiments, the attributes of the human body image may include area, grayscale value of pixels or other attributes. In some embodiments, when the attribute of the human body is area, the human body image is segmented for the current frame image according to a pre-trained image segmentation model, and mask images of multiple human bodies with an area greater than the set area can be obtained.

[0083] In the embodiments of the present disclosure, the image segmentation model can be implemented by a neural network. For example, the image segmentation model can be implemented by a fully convolutional neural network or other neural networks.

[0084] In the embodiments of the present disclosure, the image segmentation model can be predetermined according to actual needs, which include but are not limited to time consumption requirements, accuracy requirements, etc.; that is, different image segmentation models can be set according to different actual needs.

[0085] It should be noted that the above-described contents are merely exemplary descriptions of the image segmentation model, and the embodiments of the present disclosure are not limited thereto.

[0086] In the embodiment of the present disclosure, the two-dimensional key point information may include coordinate information of the two-dimensional key point, and the coordinate information of the two-dimensional key point includes a horizontal coordinate and a vertical coordinate.

[0087] The three-dimensional key point information may include coordinate information of the three-dimensional key point, where the coordinate information of the three-dimensional key point represents the coordinate of the three-dimensional key point in the camera coordinate system, wherein the camera coordinate system represents a three-dimensional rectangular coordinate system with the focusing center of the image acquisition device as the origin and the optical axis of the image acquisition device as the Z axis, and the X-axis and Y-axis of the camera coordinate system are two mutually perpendicular coordinate axes of the image plane.

[0088] In some embodiments, after determining the two-dimensional key point information, the three-dimensional key point corresponding to the two-dimensional key point can be determined based on the two-dimensional key point information, and the coordinate information of the three-dimensional key point can be determined; exemplarily, a key point conversion model can be pre-trained, and the key point conversion model is used to realize the conversion of two-dimensional key points to three-dimensional key points; in this way, after obtaining the trained key point conversion model, the coordinate information of the two-dimensional key point can be input into the trained key point conversion model to obtain the three-dimensional key point corresponding to the two-dimensional key point and the coordinate information of the three-dimensional key point. It should be noted that the above-mentioned content is only an exemplary description of obtaining the coordinate information of the three-dimensional key, and the embodiments of the present disclosure are not limited to this.

[0089] In the disclosed embodiments, the network structure of the key point conversion model is not limited. For example, the key point conversion model can be a temporal convolutional network or a non-temporal fully connected network. The network structure of the key point conversion model can be predetermined according to actual application requirements.

[0090] In some embodiments, when the above-mentioned at least one frame image is a multi-frame image, human body key points can be detected and tracked for at least one frame image to obtain two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame image; it can be understood that tracking human body key points based on multiple frame images is conducive to accurately obtaining two-dimensional key point information of multiple human bodies in the current frame image, and thus is conducive to obtaining accurate three-dimensional key point information.

[0091] In some embodiments, when at least one of the above-mentioned frames is a continuous frame image, the continuous frame images can be used to detect and track human body key points to obtain two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame image; it can be understood that tracking human body key points based on continuous frame images is conducive to further accurately deriving two-dimensional key point information of multiple human bodies in the current frame image, and thus is conducive to obtaining accurate three-dimensional key point information.

[0092] Step 203: Determine depth detection results of the multiple human bodies in the current frame image according to the two-dimensional key point information and the three-dimensional key point information of the multiple human bodies in the current frame image, and the mask images of the multiple human bodies.

[0093] In practical applications, the above steps 201 to 203 can be implemented based on a processor of an electronic device, and the above processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device that implements the above processor function can also be other, and the embodiments of the present disclosure are not limited.

[0094] It can be seen that the embodiments of the present disclosure can combine the human body mask images of multiple human bodies and the two-dimensional key points and three-dimensional key information of multiple human bodies to determine the depth detection results of multiple human bodies, without the need to obtain the depth information of the human bodies in the image through special hardware devices such as three-dimensional depth cameras. Therefore, the embodiments of the present disclosure can realize depth detection of multiple human bodies in the image without relying on special hardware devices such as three-dimensional depth cameras, and can be applied to scenes such as AR interaction and virtual photography.

[0095] Furthermore, the embodiments of the present disclosure can obtain the depth information between each pixel of the human body and the camera, rather than estimating a single depth for each pixel of the human body. The obtained depth information is relatively rich and can be applied to multiple scenarios. For example, the application scope of the embodiments of the present disclosure includes but is not limited to: three-dimensional reconstruction and presentation of a dynamic human body in three-dimensional human body reconstruction; occlusion display of a human body and a virtual scene in augmented reality applications; interaction between a human body and a reconstructed scene in augmented reality applications, etc. Furthermore, the embodiments of the present disclosure do not directly estimate the depth information of human body pixels based on an image matching algorithm of continuous frames, but use the two-dimensional key point information and three-dimensional key point information of the human body to determine the depth information of human body pixels. Compared with a solution that realizes depth information estimation based on an image matching algorithm of continuous frames, it reduces the consumption of time resources and computing resources, and achieves a balance between the estimation accuracy of depth information and the time consumption of determining depth information.

[0096] In some embodiments, at least one frame of image captured by the above-mentioned image acquisition device is an RGB image; it can be seen that the embodiment of the present disclosure can realize depth detection of multiple human bodies based on easily acquired RGB images, and has the characteristic of being easy to implement.

[0097] In some embodiments, the two-dimensional key point information of each human body in the multiple human bodies can be matched with the mask image of each human body in the mask images of the multiple human bodies to obtain the two-dimensional key point information belonging to each human body respectively; then, the depth detection result of each human body in the current frame image is determined according to the three-dimensional key point information corresponding to the two-dimensional key point information belonging to each human body respectively.

[0098] It can be seen that the embodiment of the present disclosure can directly obtain the two-dimensional key point information of each human body by matching the two-dimensional key point information of multiple human bodies in the current frame image with the mask image of each human body, and then determine the depth detection result of each human body.

[0099] In the disclosed embodiment, the two-dimensional key point information is the two-dimensional key point representing the human skeleton, and the three-dimensional key point information is the three-dimensional key point representing the human skeleton.

[0100] The two-dimensional key points of the human skeleton are used to characterize the key positions of the human body in the image plane, and the key positions of the human body include but are not limited to the facial features, neck, shoulders, elbows, hands, hips, knees, feet, etc. The key positions of the human body can be pre-set according to the actual situation; for example, refer to Figure 3 , the two-dimensional key points of the human skeleton can represent 14 key positions of the human body or 17 key positions of the human body, Figure 3 In the figure, hollow circles represent 14 key positions of the human body, and hollow circles and solid dots together represent 17 key positions of the human body.

[0101] It can be seen that the embodiments of the present disclosure can derive the two-dimensional key points of each human skeleton, and determine the depth detection result of each human body based on the two-dimensional key points of each human skeleton. Since the depth detection of different human bodies in the image depends on the two-dimensional key points of the skeletons of different human bodies, the correlation between the two-dimensional key points of the skeletons of different human bodies is relatively small. Therefore, the embodiments of the present disclosure can realize the depth detection of multiple human bodies in the image.

[0102] In some embodiments, among the two-dimensional key point information of the above-mentioned multiple human bodies, the two-dimensional key point information of a human body whose position overlap with the mask image of each human body reaches a set value can be used as the two-dimensional key point information of each human body.

[0103] Here, the set value can be a numerical value pre-set according to the actual application scenario, for example, the set value is between 80% and 90%; in the embodiment of the present disclosure, the overlap between the two-dimensional key point information of each human body and the human body mask image can be determined based on the coordinate information of the two-dimensional key point of each human body among multiple human bodies and the position information of the human body mask image.

[0104] In some embodiments, for any mask image of a human body, if the position overlap of two-dimensional key point information of multiple human bodies with the mask image reaches a set value, the two-dimensional key point information of a human body with the highest position overlap with the mask image can be selected from the two-dimensional key point information of the multiple human bodies.

[0105] It can be seen that in the disclosed embodiment, the two-dimensional key point information of each human body can be directly determined based on the position overlap between the two-dimensional key point information and the human body mask image, which is conducive to accurately obtaining the two-dimensional key point information of each human body.

[0106] In some embodiments, the two-dimensional key point information of multiple human bodies in the above-mentioned current frame image can be optimized to obtain the two-dimensional key point information of multiple human bodies after optimization; then, among the two-dimensional key point information of multiple human bodies after optimization, the two-dimensional key point information of a human body whose position overlap with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

[0107] For the implementation method of optimizing the two-dimensional key point information of multiple human bodies in the current frame image to obtain the two-dimensional key point information of multiple human bodies after optimization, exemplarily, when at least one frame image also includes a historical frame image, the two-dimensional key point information of multiple human bodies in the current frame image and the two-dimensional key point information of multiple human bodies in the historical frame image can be processed to obtain the two-dimensional key point information of multiple human bodies after optimization.

[0108] In some embodiments, the two-dimensional key point information of multiple human bodies in the current frame image and the two-dimensional key point information of multiple human bodies in the historical frame images can be subjected to time series filtering processing to obtain the two-dimensional key points of multiple human bodies after filtering processing; the method of time series filtering processing includes but is not limited to time series low-pass filtering and time series extended Kalman filtering; in other embodiments, the two-dimensional key point information of multiple human bodies in the current frame image and the two-dimensional key point information of multiple human bodies in the historical frame images can be subjected to skeleton limb length optimization processing to obtain the two-dimensional key point information of multiple human bodies after filtering processing.

[0109] It can be understood that optimizing the two-dimensional key point information of multiple human bodies in the current frame image by combining the two-dimensional key point information of multiple human bodies in the historical frame image is beneficial to improving the temporal stability of the two-dimensional key point information, and further to improving the temporal stability of human body depth detection.

[0110] For the implementation method of determining the depth detection result of each human body in the current frame image based on the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body, exemplarily, the coordinate information of the three-dimensional key point information corresponding to the two-dimensional key point information of each human body can be determined; based on the coordinate information of the three-dimensional key point, the depth information of the two-dimensional key point of each human body can be determined; the depth information of the two-dimensional key point of each human body is interpolated to obtain the depth information of the first pixel point in the mask image of each human body; wherein the first pixel point represents any pixel point in the mask image of each human body except the pixel point overlapping with the two-dimensional key point position.

[0111] Exemplarily, since the two-dimensional key points correspond to the three-dimensional key points, the coordinate information of the three-dimensional key point information of each human body can be used as the depth information of the two-dimensional key point of each human body. Here, the depth information of the two-dimensional key point represents: the depth information of the pixel point overlapping with the two-dimensional key point position.

[0112] If any pixel point in the mask image of each human body is not a pixel point overlapping with the position of the two-dimensional key point, then any of the above pixel points can be considered as the first pixel point. At this time, the depth information of the two-dimensional key point of each human body can be interpolated to obtain the depth information of the first pixel point in the mask image of each human body.

[0113] Interpolation is an important method for approximating discrete functions. It can be used to estimate the approximate value of a function at other points based on the value of the function at a finite number of points. In some embodiments, the depth information of the complete pixel points in the mask image of each human body can be obtained based on the interpolation processing method under the preset spatial continuity constraint.

[0114] In some embodiments, after the depth information of each pixel point in the mask image of each human body is obtained, a smoothing filtering process may be performed on the depth information of each pixel point in the mask image of each human body.

[0115] In some embodiments, after obtaining the depth information of each pixel in the mask image of each human body, a depth map of each human body can be generated based on the depth information of each pixel, and the depth map can be displayed on the display interface of the electronic device.

[0116] It can be seen that the embodiment of the present disclosure can determine the depth information for any pixel point of the mask image of each human body, and can fully realize the depth detection of each human body in the image.

[0117] For the implementation method of interpolating the depth information of each two-dimensional key point of the human body to obtain the depth information of the first pixel point in the mask image of each human body, exemplarily, a discrete function for characterizing the relationship between the pixel point position and the pixel point depth information can be determined based on the depth information of the two-dimensional key point of each human body; based on the depth information of the two-dimensional key point of each human body, the value of the discrete function at the position of the first pixel point is supplemented, and the value of the discrete function at the position of the first pixel point is determined as the depth information of the first pixel point.

[0118] In the embodiments of the present disclosure, the above-described contents merely illustrate the principles of the interpolation processing and do not limit the specific implementation methods of the interpolation processing. By way of example, the specific implementation methods of the interpolation processing include but are not limited to nearest neighbor interpolation completion, interpolation completion based on breadth-first search, etc.

[0119] It can be understood that by interpolating the depth information of each two-dimensional key point of the human body, the spatial continuity requirement of the depth information of each pixel point of the human body can be met, which is beneficial to improving the spatial continuity of the human body depth detection results.

[0120] The depth detection method of the embodiment of the present disclosure is further described below with reference to the accompanying drawings.

[0121] Figure 4 Another optional flow chart of the depth detection method according to the embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the image acquisition device can send the acquired multiple frames of images to the processor of the electronic device, where the multiple frames of images include the current frame image and the historical frame image, and the multiple frames of images are all RGB images; the processor can perform human body image segmentation on the current frame image of at least one frame of images to obtain mask images of multiple human bodies; and can also detect and track human body key points based on the multiple frames of images to obtain two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame image. After obtaining the two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame image, post-processing optimization can also be performed, and the post-processing optimization includes the above-mentioned process of performing temporal filtering processing on the two-dimensional key point information and the above-mentioned process of performing interpolation processing on the depth information of the two-dimensional key points.

[0122] After performing post-processing optimization, the depth detection results of the multiple human bodies in the current frame image are determined according to the two-dimensional key point information and three-dimensional key point information of the multiple human bodies in the current frame image, and the mask images of the multiple human bodies. A depth map of each human body is generated based on the depth detection results of the multiple human bodies in the current frame image, and the depth map can be displayed in the display interface of the electronic device to achieve human-computer interaction.

[0123] In some embodiments, the point cloud corresponding to each pixel in the depth map may also be displayed; Figure 5 A schematic diagram of a point cloud provided in an embodiment of the present disclosure, Figure 5 In the figure, the points within the human body contour represent the point cloud composed of pixel points, the bold solid dots represent the skeleton key points, and the lines between the bold solid dots represent the human body skeleton.

[0124] It can be understood that by displaying the point cloud corresponding to each pixel in the depth map, it is easy to intuitively know the position of the pixel. Further, by displaying the skeleton key points, it is helpful to intuitively understand the relationship between the pixel and the skeleton key points.

[0125] In some embodiments, after obtaining depth detection results of multiple human bodies in the current frame image, AR effects can also be displayed based on the depth detection results of the multiple human bodies.

[0126] In some embodiments, the positional relationship between each of the multiple human bodies and at least one target object in the AR scene can be determined based on the depth detection results of the multiple human bodies in the current frame image; based on the positional relationship, the combined presentation method of the multiple human bodies and the at least one target object is determined; based on the combined presentation method, the AR effect of the superposition of the multiple human bodies and the at least one target object is displayed.

[0127] Here, the target object may be an object actually existing in a real scene, and the depth information of the target object may be known, or may be information determined based on the photographed data of the target object; the target object may also be a pre-set virtual object, and the depth information of the virtual object is predetermined.

[0128] In one embodiment, the positional relationship between multiple human bodies and at least one target object in the AR scene, as well as the positional relationship between multiple human bodies can be determined based on the depth detection results of multiple human bodies and the depth information of the target object; exemplarily, the positional relationship between each human body and the target object in the AR scene can be the following situations: 1) the human body is closer to the image acquisition device than the target object, 2) the target object is closer to the image acquisition device than the human body, 3) the human body is located on the right, left, upper or lower side of the target object, 4) a part of the human body is closer to the image acquisition device than the target object, and the other part is farther from the image acquisition device than the target object; the positional relationship between multiple human bodies can be the following situations: 1) one human body is closer to the image acquisition device relative to another human body, 2) one human body is located on the side, left, upper or lower side of another human body, 3) part of a human body is closer to the image acquisition device than another human body, and other parts are farther from the image acquisition device than another human body; it should be noted that the above is only an exemplary description of the positional relationship between multiple human bodies and at least one target object in the AR scene, and the embodiments of the present disclosure are not limited to this.

[0129] After determining the positional relationship between multiple human bodies and at least one target object in the AR scene, the combined presentation method of the multiple human bodies and the at least one target object can be determined so that the combined presentation method reflects the above-mentioned positional relationship. In this way, based on the combined presentation method, the AR effect of superimposing multiple human bodies and at least one target object is displayed, which is conducive to improving the AR display effect.

[0130] Based on the depth detection method described in the foregoing embodiment, the embodiment of the present disclosure further provides a depth detection device, which can be located in the electronic device described above.

[0131] Figure 6 A schematic diagram of an optional structure of the depth detection device provided in the embodiment of the present disclosure is shown as follows: Figure 6 As shown, the depth detection device 600 may include:

[0132] An acquisition module 601 is used to acquire at least one frame of image acquired by an image acquisition device, wherein the at least one frame of image includes a current frame of image;

[0133] The processing module 602 is used to segment the human body image of the current frame image to obtain a plurality of human body mask images; and detect the human body key points of the at least one frame image to obtain the two-dimensional key point information and the three-dimensional key point information of the plurality of human bodies in the current frame image;

[0134] The detection module 603 is used to determine the depth detection results of the multiple human bodies in the current frame image according to the two-dimensional key point information and the three-dimensional key point information of the multiple human bodies in the current frame image and the mask images of the multiple human bodies.

[0135] In some embodiments of the present disclosure, the detection module 603 is used to determine the depth detection results of the multiple human bodies in the current frame image according to the two-dimensional key point information and the three-dimensional key point information of the multiple human bodies in the current frame image, and the mask images of the multiple human bodies, including:

[0136] By matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, the two-dimensional key point information respectively belonging to each human body is obtained;

[0137] The depth detection result of each human body in the current frame image is determined according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body.

[0138] In some embodiments of the present disclosure, the detection module 603 is used to obtain two-dimensional key point information belonging to each human body by matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, including:

[0139] Among the two-dimensional key point information of the plurality of human bodies, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

[0140] In some embodiments of the present disclosure, the detection module 603 is used to determine the depth detection result of each human body in the current frame image according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body, including:

[0141] Determine the coordinate information of the three-dimensional key point information corresponding to the two-dimensional key point information of each human body; determine the depth information of the two-dimensional key point of each human body according to the coordinate information of the three-dimensional key point; interpolate the depth information of the two-dimensional key point of each human body to obtain the depth information of the first pixel point in the mask image of each human body; wherein the first pixel point represents any pixel point in the mask image of each human body except the pixel point overlapping with the position of the two-dimensional key point.

[0142] In some embodiments of the present disclosure, the detection module 603 is used to obtain two-dimensional key point information belonging to each human body by matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, including:

[0143] Optimizing the two-dimensional key point information of the plurality of human bodies in the current frame image to obtain the optimized two-dimensional key point information of the plurality of human bodies;

[0144] Among the two-dimensional key point information of the plurality of human bodies after the optimization process, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

[0145] In some embodiments of the present disclosure, the detection module 603 is used to optimize the two-dimensional key point information of multiple human bodies in the current frame image to obtain the optimized two-dimensional key point information of multiple human bodies, including:

[0146] In the case where the at least one frame image also includes a historical frame image, the two-dimensional key point information of multiple human bodies in the current frame image and the two-dimensional key point information of multiple human bodies in the historical frame image are processed to obtain the two-dimensional key point information of multiple human bodies after optimization.

[0147] In some embodiments of the present disclosure, the processing module 602 is further configured to:

[0148] Determine, according to the depth detection results of the multiple human bodies in the current frame image, a positional relationship between the multiple human bodies and at least one target object in the AR scene;

[0149] Based on the positional relationship, determining a combined presentation mode of the plurality of human bodies and the at least one target object;

[0150] Based on the combined presentation method, an AR effect of superimposing the multiple human bodies and the at least one target object is displayed.

[0151] In some embodiments of the present disclosure, at least one frame of image captured by the image acquisition device is an RGB image.

[0152] In some embodiments of the present disclosure, the two-dimensional key point information is two-dimensional key points representing a human skeleton, and the three-dimensional key point information is three-dimensional key points representing a human skeleton.

[0153] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0154] It should be noted that in the embodiments of the present disclosure, if the above-mentioned display method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to enable a computer device (which can be a terminal, server, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific combination of hardware and software.

[0155] Correspondingly, an embodiment of the present disclosure further provides a computer program product, which includes computer executable instructions, and the computer executable instructions are used to implement the depth detection method provided by the embodiment of the present disclosure.

[0156] Accordingly, an embodiment of the present disclosure further provides a computer storage medium, on which computer executable instructions are stored, and the computer executable instructions are used to implement the depth detection method provided in the above embodiment.

[0157] The present disclosure also provides an electronic device, Figure 7 A schematic diagram of an optional structure of an electronic device provided in an embodiment of the present disclosure, such as Figure 7 As shown, the electronic device 700 includes:

[0158] Memory 701, used to store executable instructions;

[0159] The processor 702 is configured to execute the executable instructions stored in the memory to implement any one of the above-mentioned depth detection methods.

[0160] The memory 701 is configured to store computer programs and applications used by the processor 702, and can also cache data to be processed or processed by the processor 702 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0161] When the processor 702 executes the program, any one of the above-mentioned depth detection methods is implemented.

[0162] The processor 702 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understandable that the electronic device implementing the functions of the processor may also be other, which is not limited in the embodiments of the present disclosure.

[0163] The above-mentioned computer-readable storage medium / memory can be ROM, programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic random access memory (Ferromagnetic Random Access Memory, FRAM), flash memory (Flash Memory), magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM) and other memories; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0164] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0165] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0166] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0167] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present disclosure.

[0168] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0169] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for the automatic test line of the device to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0170] The methods disclosed in several method embodiments provided in the present disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0171] The features disclosed in several method or device embodiments provided in the present disclosure may be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0172] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A depth detection method, characterized in that: The method comprises: Acquire at least one frame of image captured by an image acquisition device, wherein the at least one frame of image includes a current frame of image; Performing human body image segmentation on the current frame image to obtain multiple human body mask images; and performing human body key point detection on the at least one frame image to obtain two-dimensional key point information and three-dimensional key point information of multiple human bodies in the current frame image; the three-dimensional key point information is obtained by performing key point conversion on the two-dimensional key point information based on a key point conversion model; By matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies, the two-dimensional key point information respectively belonging to each human body is obtained; The depth detection result of each human body in the current frame image is determined according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body.

2. The method according to claim 1, characterized in that The method of obtaining two-dimensional key point information belonging to each human body by matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies comprises: Among the two-dimensional key point information of the plurality of human bodies, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

3. The method according to claim 1, characterized in that The step of determining the depth detection result of each human body in the current frame image according to the three-dimensional key point information corresponding to the two-dimensional key point information respectively belonging to each human body comprises: Determine coordinate information of the three-dimensional key point information corresponding to the two-dimensional key point information of each human body; Determining the depth information of each two-dimensional key point of the human body according to the coordinate information of the three-dimensional key point; The depth information of the two-dimensional key point of each human body is interpolated to obtain the depth information of the first pixel point in the mask image of each human body; wherein the first pixel point represents any pixel point in the mask image of each human body except the pixel point overlapping with the position of the two-dimensional key point.

4. The method according to claim 2 or 3, characterized in that: The method of obtaining two-dimensional key point information belonging to each human body by matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies comprises: Optimizing the two-dimensional key point information of the plurality of human bodies in the current frame image to obtain the optimized two-dimensional key point information of the plurality of human bodies; Among the two-dimensional key point information of the plurality of human bodies after the optimization process, the two-dimensional key point information of a human body whose position overlap degree with the mask image of each human body reaches a set value is used as the two-dimensional key point information of each human body.

5. The method according to claim 4, characterized in that The optimizing process of the two-dimensional key point information of the multiple human bodies in the current frame image to obtain the optimized two-dimensional key point information of the multiple human bodies includes: In the case where the at least one frame image also includes a historical frame image, the two-dimensional key point information of multiple human bodies in the current frame image and the two-dimensional key point information of multiple human bodies in the historical frame image are processed to obtain the two-dimensional key point information of multiple human bodies after optimization.

6. The method according to claim 1, characterized in that The method further comprises: Determine, according to the depth detection results of the multiple human bodies in the current frame image, a positional relationship between the multiple human bodies and at least one target object in an augmented reality (AR) scene; Based on the positional relationship, determining a combined presentation mode of the plurality of human bodies and the at least one target object; Based on the combined presentation method, an AR effect of superimposing the multiple human bodies and the at least one target object is displayed.

7. The method according to claim 1, characterized in that At least one frame of image captured by the image capture device is a red, green, and blue (RGB) image.

8. The method according to claim 1, characterized in that The two-dimensional key point information is the two-dimensional key points representing the human skeleton, and the three-dimensional key point information is the three-dimensional key points representing the human skeleton.

9. A depth detection device, characterized in that: The device comprises: An acquisition module, used to acquire at least one frame of image acquired by an image acquisition device, wherein the at least one frame of image includes a current frame of image; A processing module, configured to segment the human body image of the current frame image to obtain a plurality of human body mask images; and detect human body key points of the at least one frame image to obtain two-dimensional key point information and three-dimensional key point information of the plurality of human bodies in the current frame image; the three-dimensional key point information is obtained by performing key point conversion on the two-dimensional key point information based on a key point conversion model; A detection module is used to obtain two-dimensional key point information belonging to each human body by matching the two-dimensional key point information of each human body in the multiple human bodies with the mask image of each human body in the mask images of the multiple human bodies; and determine the depth detection result of each human body in the current frame image according to the three-dimensional key point information corresponding to the two-dimensional key point information belonging to each human body.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to execute the executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when executed by a processor, the method described in any one of claims 1 to 8 is implemented.