Social relation identification method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a social relation identification method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the steps of obtaining human body posture key point positions through a neural network model according to an input image; estimating depth information of the input image through a depth estimation model; fusing the human body posture key point position and the depth information to obtain human body posture three-dimensional space position information; and according to the human body posture three-dimensional space position information, identifying whether a social relationship exists between the human bodies, and obtaining a social relationship identification result. According to the embodiment, the human body position and the real distance can be calculated through the two-dimensional image, and the social relation recognition effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a social relationship recognition method and device, an electronic device, and a storage medium. Background Art

[0002] Among related technologies, as deep learning technology continues to advance and is widely used in the field of computer vision, video surveillance systems can use deep learning technology to achieve intelligent recognition and analysis of multiple targets such as faces, vehicles, and behaviors.

[0003] However, the current technology for identifying social relationships in security scenarios is still in its infancy. Existing technology compresses the real three-dimensional space into a plane. It is possible that the real distance is far but the pixel distance in the image is very close, which leads to the wrong estimation of close social relationships.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a social relationship recognition method and device, an electronic device and a storage medium, aiming to improve the social relationship recognition effect by calculating the position and actual distance of the human body through a two-dimensional image.

[0006] To achieve the above objective, an embodiment of the present application provides a method for identifying social relationships, the method comprising the following steps:

[0007] The position of key points of human posture is obtained according to the input image through the neural network model;

[0008] estimating depth information of the input image by using a depth estimation model;

[0009] Fusion of the human body posture key point positions and the depth information to obtain human body posture three-dimensional spatial position information, wherein the human body posture three-dimensional spatial position information includes position information of human body key points in three-dimensional space;

[0010] Whether there is a social relationship between human bodies is identified based on the three-dimensional spatial position information of the human body postures, and a social relationship identification result is obtained.

[0011] In some embodiments, obtaining the position of key points of human posture according to the input image through the neural network model includes:

[0012] Extracting a first image feature according to the input image through a backbone network;

[0013] Obtaining a key point confidence heat map and a partial affinity vector field according to the extracted first image features through a convolutional attitude machine, wherein channels of the key point confidence heat map correspond one-to-one to key points;

[0014] Fitting the key point confidence heat map through a two-dimensional Gaussian distribution to obtain the key point position, wherein the key point position includes the center position of the Gaussian distribution;

[0015] The key point positions are grouped according to the partial affinity vector field using a bipartite graph matching algorithm to obtain the key point positions of the human body posture.

[0016] In some embodiments, estimating the depth information of the input image by using a depth estimation model includes:

[0017] Extracting a second image feature according to the input image by an encoder;

[0018] Upsampling the second image feature through a decoder to obtain a multi-scale feature;

[0019] The multi-scale features are fused through a convolutional network to generate a depth map, where the depth map includes the depth information.

[0020] In some embodiments, the identifying whether there is a social relationship between human bodies according to the three-dimensional spatial position information of human body postures to obtain a social relationship identification result includes:

[0021] Calculating the personal privacy space according to the three-dimensional spatial position information of the human body posture;

[0022] Whether there is a social relationship between human bodies is identified through the three-dimensional spatial position information of the human body posture and the personal privacy space, and a social relationship identification result is obtained.

[0023] In some embodiments, the identifying whether there is a social relationship between human bodies through the three-dimensional spatial position information of human body posture and the personal privacy space to obtain a social relationship identification result includes:

[0024] Calculating the overlapping area ratio of the personal privacy space by using the three-dimensional spatial position information of the human body posture and the personal privacy space;

[0025] If the overlapping area ratio is greater than a set threshold, the social relationship recognition result is determined as the existence of a social relationship;

[0026] If the overlapping area ratio is less than or equal to the set threshold, the social relationship recognition result is determined as no social relationship exists.

[0027] In some embodiments, the three-dimensional spatial position information of the human body posture includes shoulder key points, knee key points and hip key points, and the calculating of the personal privacy space according to the three-dimensional spatial position information of the human body posture includes:

[0028] Connecting a shoulder-hip line and a knee-hip line according to the shoulder key point, the knee key point and the hip key point, wherein the shoulder-hip line is a line connecting the shoulder key point and the hip key point, and the knee-hip line is a line connecting the knee key point and the hip key point;

[0029] Calculating the angle between the shoulder-hip line and the knee-hip line;

[0030] Determine whether the angle between the connecting lines is greater than the angle threshold, if so, determine the space coefficient as the upright space coefficient, if not, determine the space coefficient as the sitting space coefficient, and the space coefficient is used to calculate the personal privacy space;

[0031] The radius of the personal privacy space area is obtained, and the personal privacy space is calculated according to the three-dimensional spatial position information of the human body posture, the radius of the personal privacy space area and the space coefficient.

[0032] In some embodiments, the personal privacy space expression is:

[0033] P={(x,y)|(x-kr) 2 +y 2 ≤(kr) 2}

[0034] Among them, P is the personal privacy space, x is the horizontal coordinate of the human body's position on the three-dimensional space plane, y is the vertical coordinate of the human body's position on the three-dimensional space plane, r is the radius of the personal privacy space area, and k is the space coefficient.

[0035] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a social relationship identification device, the device comprising:

[0036] A position extraction module is used to obtain the position of key points of human posture according to the input image through a neural network model;

[0037] A depth calculation module, used to estimate the depth information of the input image through a depth estimation model;

[0038] A fusion module, used for fusing the positions of the key points of the human body posture with the depth information to obtain the three-dimensional spatial position information of the human body posture, wherein the three-dimensional spatial position information of the human body posture includes the position information of the key points of the human body in the three-dimensional space;

[0039] The recognition module is used to recognize whether there is a social relationship between human bodies based on the three-dimensional spatial position information of the human body postures, and obtain a social relationship recognition result.

[0040] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0042] The embodiments of the present application include at least the following beneficial effects: the present application provides a social relationship recognition method and device, an electronic device and a storage medium. The scheme obtains the position of key points of human body posture according to an input image through a neural network model. The input image is a two-dimensional image, which reduces the equipment cost; the depth information of the input image is estimated through a depth estimation model; the position of key points of human body posture and the depth information are integrated to obtain the three-dimensional spatial position information of human body posture, and the real distance between human bodies in three-dimensional space can be calculated; according to the three-dimensional spatial position information of human body posture, whether there is a social relationship between human bodies is identified, and a social relationship recognition result is obtained, which is conducive to improving the social relationship recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a social relationship identification method provided in an embodiment of the present application;

[0044] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0045] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0046] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0047] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0048] Figure 6 is a flow chart of the social relationship identification method provided in the embodiment of the present application applied to the social relationship identification system;

[0049] Figure 7 is a schematic diagram of the posture estimation model structure provided in an embodiment of the present application;

[0050] Figure 8 is a schematic diagram of a personal privacy space provided in an embodiment of the present application;

[0051] Fig. 9 is a schematic diagram of the structure of a social relationship identification device provided in an embodiment of the present application;

[0052] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0054] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0055] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0058] 1) Openpose, an open source real-time multi-person pose estimation library, estimates the human pose by analyzing the key points of the human body in images or videos, identifies the various parts of the body, and infers the human posture information.

[0059] 2) Hourglass+Associative Embedding, a method for multi-person pose estimation tasks, which proposes a bottom-up method based on the Hourglass network.

[0060] 3) HigherHRNet. A neural network architecture for bottom-up human pose estimation, improved upon the High Resolution Network (HRNet).

[0061] 4) Depth-Anything is a practical project focusing on image depth estimation. It is based on a deep learning framework and can quickly evaluate image depth without requiring a specific pre-trained model. It aims to help deep learning beginners and computer vision developers master the basic principles and applications of depth estimation.

[0062] 5) Depth-FM, which transfers the prior knowledge of image synthesis from the basic diffusion model to the flow matching model, achieves more accurate and efficient depth estimation. It introduces a fast monocular depth estimation model called Flow Matching.

[0063] 6) ZeroDepth, a monocular depth estimation framework that can predict the metric scale of arbitrary test images from different domains and camera parameters.

[0064] Among related technologies, as deep learning technology continues to advance and is widely used in the field of computer vision, video surveillance systems can use deep learning technology to achieve intelligent recognition and analysis of multiple targets such as faces, vehicles, and behaviors.

[0065] However, the current technology for identifying social relationships in security scenarios is still in its infancy. Existing technology compresses the real three-dimensional space into a plane. It is possible that the real distance is far but the pixel distance in the image is very close, which leads to the wrong estimation of close social relationships.

[0066] In summary, the technical problems existing in the relevant technologies need to be improved.

[0067] In view of this, a social relationship recognition method, device, equipment and medium are provided in the embodiments of the present application. The scheme obtains the position of key points of human body posture according to the input image through a neural network model, and the input image is a two-dimensional image, which reduces the equipment cost; estimates the depth information of the input image through a depth estimation model; integrates the position of key points of human body posture with the depth information to obtain the three-dimensional spatial position information of human body posture, and can calculate the real distance between human bodies in three-dimensional space; identifies whether there is a social relationship between human bodies according to the three-dimensional spatial position information of human body posture, and obtains the social relationship recognition result, which is conducive to improving the social relationship recognition effect.

[0068] The social relationship identification method provided in the embodiment of the present application relates to the field of image processing. The social relationship identification method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements the social relationship identification method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0071] Figure 1 is an optional flowchart of the social relationship identification method provided in the embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S104.

[0072] Step S101, obtaining the key point positions of human body posture according to the input image through a neural network model;

[0073] Specifically, the input image is a two-dimensional image.

[0074] Optionally, the present application does not specifically limit the method for acquiring the input image, and can be flexibly selected in combination with actual recognition needs. For example, the input image can be acquired by a monocular camera, or image data can be captured from video data.

[0075] In some embodiments, the input image is preprocessed. Exemplarily, the image preprocessing includes but is not limited to resizing, sharpness adjustment, and normalization.

[0076] Furthermore, the input image is extracted through the neural network model to obtain the position of the key points of the human body posture. Among them, the neural network model is used to estimate the human body posture, and the top-down ("top-down") and bottom-up ("bottom-up") human body posture estimation algorithms can be selected. Exemplarily, the bottom-up algorithms of Openpose, Hourglass+AssociativeEmbedding and HigherHRNet can be used.

[0077] In some embodiments, a first image feature is extracted according to an input image through a backbone network; a key point confidence heat map and a partial affinity vector field are obtained according to the extracted first image feature through a convolutional posture machine, and the channels of the key point confidence heat map correspond one-to-one to the key points; the key point confidence heat map is fitted by a two-dimensional Gaussian distribution to obtain the key point position, and the key point position includes the center position of the Gaussian distribution; the key point positions are grouped by a bipartite graph matching algorithm to obtain the key point positions of the human body posture.

[0078] In this embodiment, the positions of key points of human posture are obtained according to the input image through a neural network model, providing basic data for subsequent three-dimensional reconstruction and social relationship analysis, reducing the requirements for image acquisition equipment and thereby reducing costs.

[0079] Step S102: Estimate the depth information of the input image using a depth estimation model.

[0080] Specifically, the depth estimation model includes an encoder and a decoder.

[0081] Optionally, the depth estimation model includes models such as depth-anything, depth-fm and zerodepth.

[0082] In some embodiments, the encoder extracts the second image features according to the input image; the decoder upsamples the second image features to obtain multi-scale features; and the convolutional network fuses the multi-scale features to generate a depth map, wherein the depth map includes depth information.

[0083] Furthermore, for the missing parts in the depth map (such as reflective surfaces, transparent objects, etc.), the depth map restoration technology is used to improve the quality of the depth map.

[0084] In this embodiment, the depth information of the input image is estimated by using a depth estimation model, which is beneficial to the subsequent three-dimensional space position calculation.

[0085] Step S103, fusing the key point positions of the human body posture with the depth information to obtain the three-dimensional spatial position information of the human body posture.

[0086] Specifically, the three-dimensional spatial position information of the human body posture includes the position information of the key points of the human body in the three-dimensional space;

[0087] It can be understood that the depth value of the depth map corresponding to the position of the key point of the human body on the image is the depth value of the human body in the three-dimensional space.

[0088] Optionally, for each key point, the three-dimensional coordinates of the point are calculated by combining its two-dimensional position in the image and the depth information in the depth map.

[0089] Furthermore, based on the depth map and the positions of key points of human body posture, the least square method or Kalman filter algorithm is used to optimize the accuracy of three-dimensional position calculation.

[0090] In some embodiments, the positions of key points of human body posture are fused with depth information to obtain the position information of all key points of the human body in three-dimensional space (ie, the three-dimensional space position information of the human body posture).

[0091] In this embodiment, the key point positions of human body postures are integrated with depth information to obtain the three-dimensional spatial position information of human body postures, and the human body postures in two-dimensional images are expanded to three-dimensional space, which is beneficial to calculate the real distance and improve the social relationship recognition effect.

[0092] Step S104, identifying whether there is a social relationship between human bodies based on the three-dimensional spatial position information of human body postures, and obtaining a social relationship identification result.

[0093] In some embodiments, the personal privacy space is calculated based on the three-dimensional spatial position information of the human body posture; the three-dimensional spatial position information of the human body posture and the personal privacy space are used to identify whether there is a social relationship between human bodies, and obtain a social relationship recognition result.

[0094] In some embodiments, the radius and space coefficient of the personal privacy space area are obtained, and the personal privacy space is calculated according to the three-dimensional spatial position information of the human body posture, the radius and space coefficient of the personal privacy space area. The size of the personal privacy space is determined by the radius and space coefficient of the personal privacy space, and the space coefficient is used to calculate the personal privacy space. The size of the personal privacy space can be adjusted according to different space coefficients.

[0095] It is understandable that the present application does not make any specific limitation on the radius of the personal privacy space area, and it can be flexibly selected in combination with actual identification needs. For example, the radius of the personal privacy space area can be pre-configured or input by the user in real time.

[0096] Furthermore, the present application does not specifically limit the spatial coefficient, and it can be flexibly selected in combination with actual recognition needs. For example, the spatial coefficient can be pre-configured, or input by the user in real time, and spatial coefficients of different postures can be applied according to different postures of the human body.

[0097] Specifically, the expression of personal privacy space is:

[0098] P={(x,y)|(x-kr) 2 +y 2 ≤(kr) 2}

[0099] Among them, P is the personal privacy space area, x is the horizontal coordinate of the human body's position on the three-dimensional space plane, y is the vertical coordinate of the human body's position on the three-dimensional space plane, r is the radius of the personal privacy space area, and k is the space coefficient.

[0100] Furthermore, the overlapping area ratio of the personal privacy space is calculated by the three-dimensional spatial position information of the human body posture and the personal privacy space; if the overlapping area ratio is greater than the set threshold, the social relationship recognition result is determined as the existence of a social relationship; if the overlapping area ratio is less than or equal to the set threshold, the social relationship recognition result is determined as the absence of a social relationship.

[0101] In other embodiments, by analyzing the relative positions of key points of human bodies (such as body orientation, arm position), it is possible to identify whether there is an interactive behavior between human bodies, and further identify whether there is a social relationship between human bodies. For example, the interactive behavior includes shaking hands, hugging, and talking.

[0102] In this embodiment, whether there is a social relationship between human bodies is identified based on the three-dimensional spatial position information of human body postures, and a social relationship identification result is obtained, which is conducive to improving the social relationship identification effect.

[0103] In steps S101 to S104 shown in the embodiment of the present application, the positions of key points of human posture are obtained according to the input image through a neural network model, and the input image is a two-dimensional image, which reduces the equipment cost; the depth information of the input image is estimated through a depth estimation model; the positions of key points of human posture and the depth information are integrated to obtain the three-dimensional spatial position information of the human posture, and the real distance between human bodies in the three-dimensional space can be calculated; according to the three-dimensional spatial position information of human posture, whether there is a social relationship between human bodies is identified, and the social relationship recognition result is obtained, which is conducive to improving the social relationship recognition effect.

[0104] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S204:

[0105] Step S201, extracting a first image feature according to an input image through a backbone network;

[0106] In step S201 of some embodiments, the input image is input into a backbone network to extract first image features. Exemplarily, the backbone network includes but is not limited to network structures such as GG, ResNet, and mobilenet.

[0107] Step S202, obtaining a key point confidence heat map and a partial affinity vector field according to the extracted first image features through a convolutional attitude machine.

[0108] Specifically, the channels of the keypoint confidence heatmap correspond one-to-one to the keypoints, and the partial affinity vector field is a vector field used to represent the connection relationship between different body parts.

[0109] In step S202 of some embodiments, the extracted first image features are input into a convolutional gesture machine to obtain key point confidence heatmaps and partial affinity vector fields.

[0110] It is understandable that each channel of the key point confidence heat map corresponds to a key point. For example, the input image uses the coco key point dataset, which contains 17 key points (including important key points of the human body such as shoulders, hips, knees, ankles, etc.), and the corresponding key point confidence heat map has 17 channels.

[0111] Step S203, fitting the key point confidence heat map through a two-dimensional Gaussian distribution to obtain the key point position.

[0112] Specifically, the key point locations include the center locations of Gaussian distributions;

[0113] In step S203 of some embodiments, key point refinement is performed using two-dimensional Gaussian distribution.

[0114] Among them, a two-dimensional Gaussian distribution is used to fit the Heatmaps of each key point, and the center position of the Gaussian distribution is taken as the final position of the key point.

[0115] Step S204, grouping the key point positions according to the partial affinity vector field by using a bipartite graph matching algorithm to obtain the key point positions of the human body posture.

[0116] It is understandable that the key point confidence heat map can be used to obtain the positions of all key points, but it cannot determine which key points belong to the same human body.

[0117] In step S204 of some embodiments, it is necessary to group the key point positions using a bipartite graph matching algorithm based on the partial affinity vector field to obtain the key point positions of the human body posture.

[0118] Exemplarily, the bipartite graph matching algorithm uses the Hungarian algorithm to match which key points belong to the same human body and group them into a complete human body pose.

[0119] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S303:

[0120] Step S301: extracting second image features according to an input image through an encoder.

[0121] In step S301 of some embodiments, the input image is divided into multiple image patches according to a certain size, and the image patches are embedded into a vector space (embedding), and the obtained vectors are input into an encoder to extract the second image features.

[0122] The embedding of image patches into the vector space includes flattening each image patch and converting each image patch into a vector through an embedding layer.

[0123] Exemplarily, the encoder may adopt a Vision Transformer (ViT) structure.

[0124] Step S302: up-sample the second image feature through a decoder to obtain a multi-scale feature.

[0125] In step S302 of some embodiments, the second image feature is upsampled by a decoder to improve the feature resolution and obtain a multi-scale feature.

[0126] Exemplarily, the decoder adopts a multi-layer upsampling convolutional network.

[0127] Optionally, the decoder uses an interpolation algorithm or deconvolution of multiple features for upsampling.

[0128] Step S303, generating a depth map by fusing multi-scale features through a convolutional network.

[0129] Specifically, the depth map includes depth information (ie, depth values).

[0130] Optionally, the multi-scale features are fused through a convolutional network in at least one of skip connection, feature addition, feature concatenation, and attention mechanism. After the multi-scale features are fused, subsequent layers of the network predict depth values.

[0131] Furthermore, a depth map is generated through a convolutional layer, where a depth value is output for each pixel position.

[0132] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S402:

[0133] Step S401, calculating the personal privacy space according to the three-dimensional spatial position information of the human body posture.

[0134] Specifically, the three-dimensional spatial position information of the human body posture includes the horizontal coordinate of the human body's position on the three-dimensional spatial ground plane and the vertical coordinate of the human body's position on the three-dimensional spatial ground plane.

[0135] In step S401 of some embodiments, the shoulder-hip line and the knee-hip line are connected according to the shoulder key points, the knee key points and the hip key points; the line angle between the shoulder-hip line and the knee-hip line is calculated; it is determined whether the line angle is greater than the angle threshold, if so, the space coefficient is determined as the upright space coefficient, otherwise the space coefficient is determined as the sitting space coefficient; the radius of the personal privacy space area is obtained, and the personal privacy space is calculated according to the three-dimensional spatial position information of the human body posture, the personal privacy space area radius and the space coefficient. Among them, the shoulder-hip line is the line between the shoulder key point and the hip key point, and the knee-hip line is the line between the knee key point and the hip key point.

[0136] Specifically, the expression of personal privacy space is:

[0137] P={(x,y)|(x-kr) 2 +y 2 ≤(kr) 2}

[0138] Among them, P is the personal privacy space, x is the horizontal coordinate of the human body's position on the three-dimensional space plane, y is the vertical coordinate of the human body's position on the three-dimensional space plane, r is the radius of the personal privacy space area, and k is the space coefficient.

[0139] Step S402, identifying whether there is a social relationship between human bodies through the three-dimensional spatial position information of human body postures and the personal privacy space, and obtaining a social relationship identification result.

[0140] In step S402 of some embodiments, the overlapping area ratio of the personal privacy space is calculated by using the three-dimensional spatial position information of the human body posture and the personal privacy space; if the overlapping area ratio is greater than a set threshold, the social relationship recognition result is determined as the existence of a social relationship; if the overlapping area ratio is less than or equal to the set threshold, the social relationship recognition result is determined as the absence of a social relationship.

[0141] It is understandable that if the three-dimensional spatial position information of the human body posture includes the position information of two people, then whether a social relationship exists is determined based on the overlapping area ratio of the personal privacy space between the two people. If the three-dimensional spatial position information of the human body posture includes the position information of more than two people, then whether a social relationship exists is determined based on the overlapping area ratio of the personal privacy space between each two people and the transitivity of the social relationship.

[0142] For example, the privacy spaces of person A and person B have a large overlap, the privacy spaces of person B and person C have a large overlap, and the privacy spaces of person A and person C have no overlap or very little overlap. It can still be determined that person A and person C have a social relationship.

[0143] See also Figure 5 In some embodiments, step S401 may also include but is not limited to steps S501 to S504:

[0144] Step S501, connecting the shoulder-hip line and the knee-hip line according to the shoulder key points, the knee key points and the hip key points.

[0145] Specifically, the three-dimensional spatial position information of the human body posture includes the shoulder key points, knee key points and hip key points of the human body, the shoulder-hip line is the line connecting the shoulder key points and the hip key points, and the knee-hip line is the line connecting the knee key points and the hip key points;

[0146] In step S501 of some embodiments, the shoulder key point and the hip key point are connected to obtain a shoulder-hip line, and the knee key point and the hip key point are connected to obtain a knee-hip line.

[0147] Step S502, calculating the angle between the shoulder-hip line and the knee-hip line.

[0148] In step S502 of some embodiments, the angle between the shoulder-hip line and the knee-hip line is calculated using a vector dot product formula.

[0149] Among them, the direction vectors of the shoulder-hip line and the knee-hip line are calculated; the dot product of the two direction vectors is calculated; the modulus of the two direction vectors is calculated; and the angle is calculated based on the dot product and the modulus using the cosine theorem.

[0150] Step S503, determining whether the connecting line angle is greater than an angle threshold, if so, determining the space coefficient as an upright space coefficient, otherwise, determining the space coefficient as a sitting space coefficient.

[0151] In step S503 of some embodiments, it is determined whether the angle between the connecting lines is greater than an angle threshold. If so, it is considered that the human body is in an upright state, and the space coefficient is determined as the upright space coefficient; otherwise, it is considered that the human body is in a sitting state, and the space coefficient is determined as the sitting space coefficient.

[0152] Step S504, obtaining the radius of the personal privacy space area, and calculating the personal privacy space according to the three-dimensional spatial position information of the human body posture, the radius of the personal privacy space area and the space coefficient.

[0153] In step S504 of some embodiments, the personal privacy space is calculated based on the horizontal and vertical coordinates of the human body on the ground plane of the three-dimensional space and the spatial coefficient. The personal privacy space expression is:

[0154] P={(x,y)|(x-kr) 2 +y 2 ≤(kr) 2}

[0155] Among them, P is the personal privacy space area, x is the horizontal coordinate of the human body's position on the three-dimensional space plane, y is the vertical coordinate of the human body's position on the three-dimensional space plane, r is the radius of the personal privacy space area, and k is the space coefficient.

[0156] Figure 6 is an optional flow chart of applying the social relationship identification method provided in the embodiment of the present application to a social relationship identification system. Figure 6 The method may include but is not limited to the following steps.

[0157] Step 1, data collection.

[0158] Specifically, the social relationship recognition system includes a data acquisition module, a visual information processing module, a social relationship recognition module, and an application function module. The data acquisition module includes an image acquisition unit, a video acquisition unit, a storage unit, and a data transmission unit.

[0159] Among them, the unit functions of the data acquisition module are as follows:

[0160] The image acquisition unit reads the camera once at a certain time interval and receives the image data from the camera.

[0161] The video acquisition unit continuously reads the camera at a certain frame rate and receives camera data. Common frame rates are 15fps and 24fps.

[0162] A storage unit, used for storing image data and video data, and managing historical data;

[0163] Data transmission interface, used to distribute the stored image data and video data to subsequent processing modules.

[0164] Step 2: Pose estimation.

[0165] Specifically, the visual information processing module is used to process the image video data obtained from the data acquisition module, and the visual information processing module includes a posture estimation unit and a depth estimation unit.

[0166] Among them, the posture estimation unit uses a neural network model to estimate the human posture. Common human posture estimation algorithms are divided into two categories: top-down and bottom-up. The bottom-up method has better performance. Common algorithms include Openpose, Hourglass+Associative Embedding, HigherHRNet, etc. Its network structure is generally composed of the following parts:

[0167] 1) Feature extraction part: Input the image to the backbone network to extract image features. Common backbone networks include VGG, ResNet, mobilenet and other network structures.

[0168] 2) Input the extracted image features into the Convolutional Pose Machines (CPM) to obtain key point confidence heatmaps and part affinity vector fields (PAFs). Each channel of the heatmap corresponds to a key point. For example, the coco key point dataset contains 17 key points (including important key points of the human body such as shoulders, hips, knees, ankles, etc.), so the corresponding heatmap has 17 channels. PAFs is a vector field used to represent the connection relationship between different body parts.

[0169] 3) Key point refinement, the key point heat map is the distribution of key point confidence on the image. A two-dimensional Gaussian distribution is used to fit the Heatmaps of each key point, and the center position of the Gaussian distribution is taken as the final position of the key point.

[0170] 4) Post-processing of posture estimation: The heatmap of key points can obtain the positions of all key points, but it cannot determine which key points belong to the same person. It is necessary to use a bipartite graph matching algorithm based on PAFs, such as the Hungarian algorithm, to match which key points belong to the same person and group them into a complete human posture.

[0171] For example, the schematic diagram of the posture estimation model structure is as follows: Figure 7 As shown in the figure, the image is not the input image, VGG / ResNet is the backbone network, which is used to extract image features; CPM (i.e., convolutional posture machine) is used to obtain key point confidence heatmaps (Heatmaps) and partial affinity vector fields (PAFs); key point refinement uses a two-dimensional Gaussian distribution to fit the Heatmaps of each key point, and the center position of the Gaussian distribution is used as the final position of the key point; posture estimation post-processing uses a bipartite graph matching algorithm based on PAFs to match which key points belong to the same human body and group them into a complete human body posture.

[0172] Step 3: Depth estimation.

[0173] Among them, the depth estimation unit uses a depth estimation model to estimate the depth information of the entire image. Common models include depth-anything, depth-fm, zerodepth, etc. Taking the depth-anything algorithm as an example, its model consists of two parts: encoder and decoder:

[0174] 1) The encoder is a Vision Transformer (ViT) structure, which divides the input image into multiple patches of a certain size, embeds the patches, and then inputs them into the transformer structure to output image features.

[0175] 2) The decoder is a multi-layer upsampling convolutional network that uses interpolation algorithms or deconvolution multiple features for upsampling to improve feature resolution and obtain multiple scale features. The multi-scale features are fused and input into the convolutional network to further generate a depth map.

[0176] Furthermore, the pose estimation and depth estimation information are integrated. The positions of the key points of the human body on the image are obtained through the pose estimation unit, and the depth information of the image is obtained through the depth unit. The depth values ​​of the depth map corresponding to the positions of the key points of the human body on the image are the depth values ​​of the human body in the three-dimensional space, and then the position information of all the key points of the human body in the three-dimensional space is obtained.

[0177] Step 4: Calculate the privacy space.

[0178] Specifically, the personal privacy space is a virtual circular area, the size of which is the distance you usually maintain when talking to others. Figure 8 As shown, the edge of the area passes through the position of the human body and is tangent to the left-right straight line of the human body.

[0179] The size of the area is determined by the input personal privacy space radius and space coefficient. The personal privacy space formula is as follows:

[0180] (x-kr) 2 +y 2 =(kr) 2

[0181] Among them, x and y are the coordinates of the human body's position on the ground plane in three-dimensional space, r is the radius of the personal privacy space area, and k is the space coefficient.

[0182] Furthermore, the personal privacy space is adjusted dynamically. People's needs for privacy space will change under different behavior patterns. For example, a larger personal privacy space is needed when standing, and the privacy space can be appropriately reduced when sitting. This module mainly calculates human behavior patterns and gives different space coefficients to different behavior patterns. The calculation method of human behavior patterns is as follows:

[0183] i) The shoulder key points and hip key points represent the position of the upper body, and the line connecting the knee key point and the hip key point represents the thigh.

[0184] ii) Calculate the angle between the shoulder-hip line and the knee-hip line. If the angle is greater than the set threshold, it means the human body is in an upright position, otherwise it means the human body is in a sitting position.

[0185] It is understandable that different spatial coefficients are configured according to different behavior patterns. By adjusting the spatial coefficient, the dynamic adjustment function of personal privacy space is realized. The privacy space is no longer static, but can adapt to user behavior and environmental changes in real time. Such a system is more intelligent and can more accurately identify and predict social relationships, while also better respecting and protecting personal privacy.

[0186] Step 5: Identify social relationships.

[0187] Optionally, social relationships are calculated through personal privacy spaces.

[0188] Among them, the social relationship between two people uses the overlapping area ratio of the privacy space as an indicator to measure the closeness of the social relationship: if the overlapping ratio is greater than the set threshold, the possibility of social interaction is greater, and it is considered that there is a social relationship between the two people; otherwise, it is considered that there is no social relationship between the two people.

[0189] Furthermore, the social relationship between multiple people is identified through the transitivity of the social relationship. For example, if the privacy spaces of person A and person B have a large overlap, and the privacy spaces of person B and person C have a large overlap, and the privacy spaces of person A and person C have no overlap or very little overlap, it can still be considered that person A and person C have a social relationship.

[0190] Step 6: Implement application functions.

[0191] Among them, the application function module is a flexible and expandable module, and different function modules can be expanded according to specific needs. This patent does not limit a specific function, and the following three functions are used as examples for explanation.

[0192] 1) In public security scenarios, monitor abnormal crowds in train stations and shopping malls to identify potential threats such as theft, fighting or terrorist activities.

[0193] 2) In smart home scenarios, monitor whether special targets such as the elderly and children are accompanied by someone, intelligently adjust home appliances, and provide safety monitoring for the elderly, children, or people who need special care.

[0194] 3) Provide more natural interactive experiences in augmented reality (AR) and virtual reality (VR) applications, such as non-verbal communication in games.

[0195] This application combines posture estimation and depth estimation to calculate the position and posture information of the human body in three-dimensional space in the monocular camera image. This method can accurately calculate the distance between human bodies in three-dimensional space. The existing method can only calculate the pixel distance in two-dimensional images. In addition, the method in this application can be applied to almost all security cameras on the market, without the need for additional new equipment, and is easier to implement.

[0196] This application calculates personal privacy space through human posture and three-dimensional spatial position information, and calculates social relationships through the degree of overlap of privacy spaces. The social relationships calculated in this way are more accurate, and the privacy space can be dynamically adjusted according to human posture to adapt to different application scenarios.

[0197] See also Fig. 9 The embodiment of the present application also provides a social relationship identification device, which can implement the above social relationship identification method, and the device includes:

[0198] A position extraction module 901 is used to obtain the position of key points of human body posture according to the input image through a neural network model;

[0199] A depth calculation module 902, configured to estimate depth information of an input image using a depth estimation model;

[0200] A fusion module 903 is used to fuse the position of the key points of the human body posture with the depth information to obtain the three-dimensional spatial position information of the human body posture, where the three-dimensional spatial position information of the human body posture includes the position information of the key points of the human body in the three-dimensional space;

[0201] The recognition module 904 is used to recognize whether there is a social relationship between human bodies according to the three-dimensional spatial position information of human body postures, and obtain a social relationship recognition result.

[0202] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0203] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above social relationship identification method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0204] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0205] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0206] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0207] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the social relationship identification method of the embodiment of this application;

[0208] Input / output interface 1003, used to implement information input and output;

[0209] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0210] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0211] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0212] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned social relationship identification method is implemented.

[0213] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0214] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0215] The social relationship recognition method, social relationship recognition device, electronic device and storage medium provided in the embodiments of the present application obtain the positions of key points of human posture according to an input image through a neural network model, and the input image is a two-dimensional image, which reduces the equipment cost; estimates the depth information of the input image through a depth estimation model; fuses the positions of key points of human posture with the depth information to obtain the three-dimensional spatial position information of the human posture, and can calculate the real distance between human bodies in the three-dimensional space; identifies whether there is a social relationship between human bodies according to the three-dimensional spatial position information of human posture, and obtains a social relationship recognition result, which is conducive to improving the social relationship recognition effect.

[0216] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0217] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0218] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0220] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0221] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0222] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0226] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A social relationship identification method, characterized in that: The method comprises the following steps: The position of key points of human posture is obtained according to the input image through the neural network model; estimating depth information of the input image by using a depth estimation model; Fusion of the human body posture key point positions and the depth information to obtain human body posture three-dimensional spatial position information, wherein the human body posture three-dimensional spatial position information includes position information of human body key points in three-dimensional space; Whether there is a social relationship between human bodies is identified based on the three-dimensional spatial position information of the human body postures, and a social relationship identification result is obtained.

2. The method according to claim 1, characterized in that The method of obtaining the key point positions of human body posture according to the input image through the neural network model includes: Extracting a first image feature according to the input image through a backbone network; Obtaining a key point confidence heat map and a partial affinity vector field according to the extracted first image features through a convolutional attitude machine, wherein channels of the key point confidence heat map correspond one-to-one to key points; Fitting the key point confidence heat map through a two-dimensional Gaussian distribution to obtain the key point position, wherein the key point position includes the center position of the Gaussian distribution; The key point positions are grouped according to the partial affinity vector field using a bipartite graph matching algorithm to obtain the key point positions of the human body posture.

3. The method according to claim 1, characterized in that The estimating the depth information of the input image by using a depth estimation model includes: Extracting a second image feature according to the input image by an encoder; Upsampling the second image feature through a decoder to obtain a multi-scale feature; The multi-scale features are fused through a convolutional network to generate a depth map, where the depth map includes the depth information.

4. The method according to claim 1, characterized in that: The identifying whether there is a social relationship between human bodies according to the three-dimensional spatial position information of human body postures to obtain a social relationship identification result includes: Calculating the personal privacy space according to the three-dimensional spatial position information of the human body posture; Whether there is a social relationship between human bodies is identified through the three-dimensional spatial position information of the human body posture and the personal privacy space, and a social relationship identification result is obtained.

5. The method according to claim 4, characterized in that The identifying whether there is a social relationship between human bodies through the three-dimensional spatial position information of human body posture and the personal privacy space to obtain a social relationship identification result includes: Calculating the overlapping area ratio of the personal privacy space by using the three-dimensional spatial position information of the human body posture and the personal privacy space; If the overlapping area ratio is greater than a set threshold, the social relationship recognition result is determined as the existence of a social relationship; If the overlapping area ratio is less than or equal to the set threshold, the social relationship recognition result is determined as no social relationship exists.

6. The method according to claim 4, characterized in that The three-dimensional spatial position information of the human body posture includes shoulder key points, knee key points and hip key points, and the calculating of the personal privacy space according to the three-dimensional spatial position information of the human body posture includes: Connecting a shoulder-hip line and a knee-hip line according to the shoulder key point, the knee key point and the hip key point, wherein the shoulder-hip line is a line connecting the shoulder key point and the hip key point, and the knee-hip line is a line connecting the knee key point and the hip key point; Calculating the angle between the shoulder-hip line and the knee-hip line; Determine whether the angle between the connecting lines is greater than the angle threshold, if so, determine the space coefficient as the upright space coefficient, if not, determine the space coefficient as the sitting space coefficient, and the space coefficient is used to calculate the personal privacy space; The radius of the personal privacy space area is obtained, and the personal privacy space is calculated according to the three-dimensional spatial position information of the human body posture, the radius of the personal privacy space area and the space coefficient.

7. The method according to claim 6, characterized in that The personal privacy space expression is: P={(x,y)|(x-kr) 2 +y 2 ≤(kr) 2 } Among them, P is the personal privacy space, x is the horizontal coordinate of the human body's position on the three-dimensional space plane, y is the vertical coordinate of the human body's position on the three-dimensional space plane, r is the radius of the personal privacy space area, and k is the space coefficient.

8. A social relationship recognition device, characterized in that: The device comprises: A position extraction module is used to obtain the position of key points of human posture according to the input image through a neural network model; A depth calculation module, used to estimate the depth information of the input image through a depth estimation model; A fusion module, used for fusing the positions of the key points of the human body posture with the depth information to obtain the three-dimensional spatial position information of the human body posture, wherein the three-dimensional spatial position information of the human body posture includes the position information of the key points of the human body in the three-dimensional space; The recognition module is used to recognize whether there is a social relationship between human bodies based on the three-dimensional spatial position information of the human body postures, and obtain a social relationship recognition result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images