Weight detection method, human body characteristic parameter detection method and device

Abstract

The present application discloses a weight detection method, a human characteristic parameter detection method, a seat adjustment method and a device in the field of artificial intelligence (AI). The weight detection method comprises: determining corresponding human body three-dimensional point cloud data according to the human body area in the two-dimensional image, and then determining the human body weight according to the human body three-dimensional point cloud data. The human body characteristic parameter detection method comprises: determining corresponding three-dimensional point cloud data according to the human body key points in the two-dimensional image, and then determining the human body characteristic parameters according to the three-dimensional point cloud data. The human body weight and / or human body characteristic parameters can be used to adjust the car seat. The method of the present application can extract human body characteristic parameters or human body weight based on the three-dimensional point cloud, thereby improving the detection accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a weight detection method, a human body characteristic parameter detection method, and a seat adjustment method and device. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras / camcorders and computers to obtain the data and information we need about the objects being photographed. Figuratively speaking, it is to equip computers with eyes (cameras / camcorders) and brains (algorithms) to replace human eyes in identifying, tracking, and measuring targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] Human body recognition is a hot topic in the field of computer vision. The research contents include: human body parameter detection, action recognition and gender recognition. Among them, human body parameter detection can be applied to a variety of scenarios, for example, in car seat adjustment. The position of the car seat and the backrest angle can be automatically adjusted according to the detected human body parameters. Therefore, the detection accuracy of human body parameters is particularly important. At present, the monocular recognition method of visible light images is usually adopted, that is, human body parameters are extracted based on two-dimensional images. This method is often affected by lighting changes, shadows, etc., and it is difficult to accurately extract human body parameters. The accuracy is low, which affects the application effect of human body parameters, for example, the experience effect of seat adjustment.

[0004] Therefore, how to accurately extract human body parameters has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a weight detection method, a human body characteristic parameter detection method, a seat adjustment method and a device, which can extract human body characteristic parameters or human body weight based on a three-dimensional point cloud (3D point cloud), thereby improving the detection accuracy.

[0006] In a first aspect, a weight detection method is provided, the method comprising: acquiring a two-dimensional image captured by a time-of-flight (TOF) camera; detecting a human body region in the two-dimensional image; converting two-dimensional data of the human body region in the two-dimensional image into three-dimensional point cloud data of the human body based on depth information corresponding to the human body region in the two-dimensional image; and determining the weight of the human body based on the three-dimensional point cloud data of the human body.

[0007] Specifically, the two-dimensional image can be segmented to obtain the human body region.

[0008] There is a mapping relationship between the imaging coordinates of the camera pixels and the coordinates of the three-dimensional point cloud, and the three-dimensional point cloud data of the human body corresponding to the human body area can be determined according to the mapping relationship.

[0009] The body weight may include a weight value or a weight range.

[0010] The layout position of the TOF camera can be set as needed to obtain the largest possible field of view so as to extract human body information over a large range.

[0011] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain the human body area, determining the human body area on a two-dimensional image and then obtaining the three-dimensional point cloud data of the human body can improve the processing accuracy and processing speed, reduce energy consumption, and effectively reduce the heating of hardware. At the same time, obtaining the human body weight based on the three-dimensional point cloud data of the human body can obtain more comprehensive and accurate human body morphology information and improve the accuracy of human body weight detection. At the same time, the use of the TOF visual system can further improve the detection accuracy, and relatively high accuracy can also be guaranteed at a long distance.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth value corresponding to the pixel in the single-channel two-dimensional image.

[0013] Specifically, the depth value corresponding to each pixel can be converted into a grayscale value, that is, the grayscale value is used to represent the depth information, and a single-channel two-dimensional image is obtained.

[0014] Optionally, the gray value of the pixel is assigned in equal proportion according to the depth value of each pixel in the TOF camera, that is, the depth value depth corresponding to the pixel is assigned in equal proportion to the gray value gray_value corresponding to the pixel.

[0015] In the embodiment of the present application, detecting the human body area in a single-channel two-dimensional image can effectively utilize the depth information, eliminate the interference of other information such as color and brightness, and obtain a more accurate human body area.

[0016] In combination with the first aspect, in certain implementations of the first aspect, detecting a human body area in a two-dimensional image includes: inputting a single-channel two-dimensional image into an image segmentation model to obtain a human body area, wherein the image segmentation model is trained based on a first training sample, the first training sample includes a first single-channel two-dimensional image and a human body area corresponding to the first single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the first single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the first single-channel two-dimensional image.

[0017] Specifically, the image segmentation model can be trained by using the first single-channel two-dimensional image as the input of the image segmentation model and the human body region corresponding to the first single-channel two-dimensional image as the target output of the image segmentation model.

[0018] The image segmentation model may be a neural network model, for example, U-net, V-net or deeplab-v3.

[0019] The existing image segmentation model is usually based on RGB images. In the embodiment of the present application, a single-channel two-dimensional image is used as a training sample for training, which can obtain a better image segmentation model and improve the accuracy of image segmentation.

[0020] In combination with the first aspect, in certain implementations of the first aspect, determining a human body's weight based on human body three-dimensional point cloud data includes: inputting the human body three-dimensional point cloud data into a weight assessment model to obtain the human body's weight, wherein the weight assessment model is trained based on a second training sample, and the second training sample includes first human body three-dimensional point cloud data and the weight corresponding to the first human body three-dimensional point cloud data.

[0021] The weight assessment model may be a neural network model. For example, the neural network model may include pointnet, etc.

[0022] The weight corresponding to the first human body 3D point cloud data can be regarded as a label corresponding to the first human body 3D point cloud data. Specifically, the first human body point cloud data is used as the input of the weight assessment model, and the corresponding human body weight is used as the target output of the weight assessment model to train the weight assessment model.

[0023] In the embodiment of the present application, the weight assessment model is trained using the human body 3D point cloud data and the corresponding weight as training samples, and the weight information is regressed from the human body 3D point cloud data to obtain the weight assessment model. In this way, the weight information can be directly determined through the human body 3D point cloud data without other calculations or conversions, thereby improving the accuracy and processing speed of the weight assessment model.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the two-dimensional image includes multiple frames of two-dimensional images, and based on the depth information corresponding to the human body area in the two-dimensional image, the two-dimensional data of the human body area in the two-dimensional image is converted into three-dimensional point cloud data of the human body, including: based on the depth information corresponding to the human body area in the multiple frames of two-dimensional images, the two-dimensional data of the human body area in the multiple frames of two-dimensional images is converted into three-dimensional point cloud data of the human body to obtain multiple frames of three-dimensional point cloud data of the human body; and the multiple frames of three-dimensional point cloud data of the human body are fused to obtain fused three-dimensional point cloud data of the human body.

[0025] In an embodiment of the present application, human body three-dimensional point cloud data is obtained by fusing multiple frames of human body three-dimensional point cloud data, which can avoid the inability to obtain complete human body three-dimensional point cloud data due to occlusion, posture, etc., and improve the accuracy of human body three-dimensional point cloud data.

[0026] In combination with the first aspect, in certain implementations of the first aspect, determining a human body weight based on human body three-dimensional point cloud data includes: uniformly refining the human body three-dimensional point cloud data to obtain sparse human body three-dimensional point cloud data; determining a human body weight based on the sparse human body three-dimensional point cloud data.

[0027] For example, the farthest point sampling method can be used to evenly refine the human body 3D point cloud data, which can reduce the number of parameters, reduce the amount of calculation, and improve the processing speed.

[0028] In a second aspect, a method for detecting human feature parameters is provided, the method comprising: acquiring a two-dimensional image captured by a time-of-flight TOF camera; detecting key points of the human body in the two-dimensional image; converting the two-dimensional data of the human body key points in the two-dimensional image into three-dimensional point cloud data based on depth information corresponding to the human body key points in the two-dimensional image, and obtaining three-dimensional point cloud data corresponding to the human body key points; determining human feature parameters based on the three-dimensional point cloud data corresponding to the human body key points, wherein the human feature parameters include: one or more of height, arm length, leg length or shoulder width.

[0029] There is a mapping relationship between the imaging coordinates of the camera pixels and the coordinates of the three-dimensional point cloud, and the three-dimensional point cloud data corresponding to the key points of the human body can be determined based on the mapping relationship.

[0030] The number and position of the key points of the human body can be set as needed. For example, the key points of the human body may include: elbow joints, shoulder joints, top of the head, ankles or wrists, etc.

[0031] Specifically, the distance between key points of the human body can be calculated based on the three-dimensional point cloud data of the key points of the human body, and the human body characteristic parameters can be determined based on the distance.

[0032] For example, the height of a person can be determined based on the distance from the top of the head to the ankle; the arm length of a person can be determined based on the distance between the wrist and the shoulder joint, etc.

[0033] The layout position of the TOF camera can be set as needed to obtain the largest possible field of view so as to extract human body information over a large range.

[0034] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain key points, key point detection on a two-dimensional image can improve processing accuracy and processing speed, reduce energy consumption, and effectively reduce hardware heating. At the same time, the human body characteristic parameters are determined based on the three-dimensional point cloud data of the key points of the human body, which improves the accuracy of human body characteristic parameter detection. At the same time, the use of a TOF visual system can further improve the detection accuracy of human body characteristic parameters, and relatively high accuracy can also be guaranteed at a long distance.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image.

[0036] Specifically, the depth value corresponding to each pixel can be converted into a grayscale value, that is, the grayscale value is used to represent the depth information, and a single-channel two-dimensional image is obtained.

[0037] Optionally, the gray value of the pixel is assigned in equal proportion according to the depth value of each pixel in the TOF camera, that is, the depth value depth corresponding to the pixel is assigned in equal proportion to the gray value gray_value corresponding to the pixel.

[0038] In the embodiment of the present application, the human body key point detection is performed using a single-channel two-dimensional image, which can effectively utilize the depth information, eliminate the interference of other information such as color and brightness, and obtain more accurate human body key points.

[0039] In combination with the second aspect, in certain implementations of the second aspect, detecting human key points in a two-dimensional image includes: inputting a single-channel two-dimensional image into a key point detection model to obtain human key points, wherein the key point detection model is trained based on a third training sample, the third training sample includes a second single-channel two-dimensional image and human key points corresponding to the second single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the second single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the second channel two-dimensional image.

[0040] The key point detection model may be a neural network model. For example, the neural network model may include openpose.

[0041] Existing key point detection models are usually based on RGB images. In the embodiments of the present application, single-channel two-dimensional images are used as training samples for training, which can obtain a better key point detection model and improve the accuracy of key point detection.

[0042] In combination with the second aspect, in certain implementation methods of the second aspect, the two-dimensional image includes multiple frames of two-dimensional images, and determining human body feature parameters based on three-dimensional point cloud data corresponding to human body key points, including: determining the three-dimensional point cloud data corresponding to human body key points in multiple frames of two-dimensional images, and obtaining three-dimensional point cloud data corresponding to multiple frames of human body key points; obtaining multiple groups of human body feature parameters based on the three-dimensional point cloud data corresponding to multiple frames of human body key points; and determining the final human vital sign parameters based on the multiple groups of human body feature parameters.

[0043] For example, n groups of human body key points are obtained based on n frames of two-dimensional images, and then the three-dimensional point cloud data corresponding to the n groups of human body key points are determined to obtain n groups of heights, and the average value of the n groups of heights is calculated, and the average value is used as the final height.

[0044] In the embodiment of the present application, by optimizing the determined human feature parameter results through multiple frames of two-dimensional images, it is possible to avoid the inability to obtain the correct key point position due to occlusion, posture, etc., thereby improving the accuracy of human feature parameter detection.

[0045] In a third aspect, a seat adjustment method is provided, the method comprising: acquiring a two-dimensional image captured by a TOF camera; detecting a human body area in the two-dimensional image; converting the two-dimensional data of the human body area in the two-dimensional image into three-dimensional point cloud data of the human body based on depth information corresponding to the human body area in the two-dimensional image; determining the human body weight according to the three-dimensional point cloud data of the human body; and adjusting the parameters of the car seat according to the human body weight.

[0046] The TOF camera can be placed outside the vehicle, so that the passenger's information can be obtained before the passenger gets on the vehicle, and the seat parameters can be adjusted in advance to improve the user experience.

[0047] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain the human body area, determining the human body area on a two-dimensional image and then obtaining the three-dimensional point cloud data of the human body can improve the processing accuracy and processing speed, reduce energy consumption, and effectively reduce hardware heating. At the same time, obtaining the human body weight based on the three-dimensional point cloud data of the human body can obtain more comprehensive and accurate human body morphological information, improve the accuracy of human body weight detection, better adjust the parameters of the car seat, and improve the user experience. At the same time, the use of the TOF visual system can further improve the detection accuracy, and can also ensure relatively high accuracy at a long distance.

[0048] In combination with the third aspect, in certain implementations of the third aspect, the method also includes: detecting human body key points in a two-dimensional image; converting the two-dimensional data of the human body key points in the two-dimensional image into three-dimensional point cloud data based on the depth information corresponding to the human body key points in the two-dimensional image, and obtaining the three-dimensional point cloud data corresponding to the human body key points; determining human body feature parameters based on the three-dimensional point cloud data corresponding to the human body key points, wherein the human body feature parameters include: one or more of height, arm length, leg length or shoulder width; adjusting the car seat parameters according to the human body weight and the human body feature parameters.

[0049] In combination with the third aspect, in certain implementations of the third aspect, the TOF camera is located at the junction of the B-pillar and the roof of the car.

[0050] The inclination angle of this position is large, which can obtain a larger field of view, so that most of the angles of boarding can be covered, and the information of passengers can be extracted more comprehensively.

[0051] In combination with the third aspect, in certain implementations of the third aspect, the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image.

[0052] In a fourth aspect, a seat adjustment method is provided, the method comprising: acquiring a two-dimensional image captured by a time-of-flight TOF camera; detecting key points of a human body in the two-dimensional image; converting the two-dimensional data of the key points of the human body in the two-dimensional image into three-dimensional point cloud data based on the depth information corresponding to the key points of the human body in the two-dimensional image, and obtaining three-dimensional point cloud data corresponding to the key points of the human body; determining human body characteristic parameters according to the three-dimensional point cloud data corresponding to the key points of the human body, wherein the human body characteristic parameters include: height, arm length, leg length or shoulder width; and adjusting the parameters of the car seat according to the human body characteristic parameters.

[0053] The TOF camera can be placed outside the vehicle, so that the passenger's information can be obtained before the passenger gets on the vehicle, and the seat parameters can be adjusted in advance to improve the user experience.

[0054] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain key points, key point detection on a two-dimensional image can improve processing accuracy and processing speed, reduce energy consumption, and effectively reduce hardware heating. At the same time, the human body characteristic parameters are determined based on the three-dimensional point cloud data of the key points of the human body, which improves the accuracy of human body characteristic parameter detection, can better adjust the parameters of the car seat, and improve the user experience. At the same time, the use of the TOF visual system can further improve the detection accuracy, and can also ensure relatively high accuracy at a long distance.

[0055] In combination with the fourth aspect, in certain implementations of the fourth aspect, the method also includes: detecting a human body area in a two-dimensional image; converting two-dimensional data of the human body area in the two-dimensional image into three-dimensional point cloud data of the human body based on depth information corresponding to the human body area in the two-dimensional image; determining the human body weight based on the three-dimensional point cloud data of the human body; and adjusting the car seat parameters based on the human body weight and human body characteristic parameters.

[0056] In combination with the fourth aspect, in certain implementations of the fourth aspect, the TOF camera is located at the junction of the B-pillar and the roof of the car.

[0057] The inclination angle of this position is large, which can obtain a larger field of view, so that most of the angles of boarding can be covered, and the information of passengers can be extracted more comprehensively.

[0058] It should be understood that the expansion, limitation, explanation and description of the relevant contents in the above-mentioned first aspect also apply to the same contents in the second aspect, the third aspect and the fourth aspect.

[0059] In a fifth aspect, a training method for a weight assessment model is provided, the method comprising: obtaining a second training sample, the second training sample comprising: first human body three-dimensional point cloud data and a weight corresponding to the first human body three-dimensional point cloud data; using the second human body three-dimensional point cloud data as an input of a weight assessment model, and using the weight corresponding to the second human body three-dimensional point cloud data as a target output of the weight assessment model for training, to obtain a trained weight assessment model.

[0060] In a sixth aspect, a weight detection device is provided, the device comprising a module or unit for executing the method in the above-mentioned first aspect and any one of the implementations of the first aspect.

[0061] In a seventh aspect, a human characteristic parameter detection device is provided, the device comprising a module or unit for executing the method in the above-mentioned second aspect and any one of the implementation modes of the second aspect.

[0062] In an eighth aspect, a seat adjustment device is provided, the device comprising a module or unit for executing the method in the third aspect and any one of the implementations of the third aspect.

[0063] In a ninth aspect, a weight detection device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the above-mentioned first aspect and any one of the implementation methods of the first aspect.

[0064] In the tenth aspect, a human feature parameter detection device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the above-mentioned second aspect and any one of the implementation methods of the second aspect.

[0065] In the eleventh aspect, a seat adjustment device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the above-mentioned third aspect and any one of the implementation methods of the third aspect.

[0066] In a twelfth aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementation modes of the above aspects.

[0067] In a thirteenth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute a method in any one of the implementation modes of the above aspects.

[0068] In a fourteenth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes a method in any one of the implementation modes in the above aspects.

[0069] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementation methods in the above aspects.

[0070] The above chip can specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a structural diagram of the system architecture provided by the embodiment of the present application;

[0072] Figure 2 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0073] Figure 3 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0074] Figure 4 A schematic flow chart of a weight detection method provided in an embodiment of the present application;

[0075] Figure 5 A schematic diagram of a method for calculating three-dimensional point cloud coordinates provided in an embodiment of the present application;

[0076] Figure 6 A schematic flow chart of a human body region segmentation method provided in an embodiment of the present application;

[0077] Figure 7 A schematic flow chart of a weight calculation method provided in an embodiment of the present application;

[0078] Figure 8 A schematic flow chart of a training method for a weight assessment model provided in an embodiment of the present application;

[0079] Fig. 9 A schematic flow chart of a human body characteristic parameter method provided in an embodiment of the present application;

[0080] Fig.10 A schematic flow chart of a method for detecting key points of a human body provided in an embodiment of the present application;

[0081] Fig.11 A schematic diagram of human body point cloud data provided in an embodiment of the present application;

[0082] Fig.12 A schematic diagram of key points of the human body provided in an embodiment of the present application;

[0083] Fig.13 A schematic flow chart of a seat adjustment method provided in an embodiment of the present application;

[0084] Fig.14 A schematic diagram of the layout position of a TOF camera provided in an embodiment of the present application;

[0085] Fig.15 A schematic flow chart of another seat adjustment method provided in an embodiment of the present application;

[0086] Fig.16 A schematic flow chart of another seat adjustment method provided in an embodiment of the present application;

[0087] Fig.17 is a schematic block diagram of a weight detection device provided in an embodiment of the present application;

[0088] Fig.18 is a schematic block diagram of a human body characteristic parameter detection device provided in an embodiment of the present application;

[0089] Fig.19 is a schematic block diagram of a seat adjustment device provided in an embodiment of the present application;

[0090] Fig. 20 is a schematic block diagram of another weight detection device provided in an embodiment of the present application;

[0091] Fig.21 is a schematic block diagram of another human characteristic parameter detection device provided in an embodiment of the present application;

[0092] Fig. 22 It is a schematic block diagram of another seat adjustment device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0094] The human characteristic parameter detection method and weight detection method provided in the embodiments of the present application can be applied to scenarios that require human characteristic parameters or human weight, such as autonomous driving. Specifically, the human characteristic parameter detection method and weight detection method in the embodiments of the present application can be applied to seat adjustment scenarios, which are briefly introduced below.

[0095] Seat adjustment:

[0096] The intelligence of car seats is an important part of future car development. Among them, seat adjustment aims to meet people's needs for car safety, comfort and intelligence by automatically adjusting the seat posture and shape. 。

[0097] The human body characteristic parameter detection method in the embodiment of the present application can obtain more accurate human body characteristic parameters such as human height, leg length, shoulder width, etc., and then based on the above human body characteristic parameters, the position of the car seat or the backrest angle and other parameters can be adjusted more accurately to better adapt to the passengers and improve the user's riding experience. Alternatively, the weight detection method in the embodiment of the present application can obtain more accurate human body weight, and then based on the above human body weight, the position of the car seat or the backrest angle and other parameters can be adjusted more accurately to better adapt to the passengers and improve the user's riding experience.

[0098] The following describes the method provided by this application from the perspectives of model training and model application:

[0099] The training method of the human feature parameter detection model provided in the embodiment of the present application and the training method of the human body weight assessment model, which are related to the processing of computer vision, can be specifically applied to data training, machine learning, deep learning and other data processing methods, and the training data (such as the first training sample or the second training sample in the present application) is symbolized and formalized intelligent information modeling, extraction, preprocessing, training, etc., and finally a trained human feature parameter detection model and a human body weight assessment model are obtained; and the human feature parameter detection method provided in the embodiment of the present application can apply the above-mentioned trained human feature parameter detection model, input the input data (such as the two-dimensional image in the present application) into the trained human feature parameter detection model, and obtain output data (such as the human feature parameters in the present application). Alternatively, the weight detection method provided in the embodiment of the present application can apply the above-mentioned trained human body weight assessment model, input the input data (such as the human three-dimensional point cloud data in the present application) into the trained human body weight assessment model, and obtain output data (such as the human body weight or human body weight range in the present application). Alternatively, the seat adjustment method provided in the embodiment of the present application can use the above-mentioned trained human feature parameter detection model and / or human body weight assessment model to input input data (such as the two-dimensional image and / or human body three-dimensional point cloud data in the present application) into the trained human feature parameter detection model and / or human body weight assessment model to obtain output data (such as the car seat parameters in the present application). It should be noted that the training method of the human body weight assessment model and the weight detection method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage. The training method of the human feature parameter detection model and the human feature parameter detection method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0100] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0101] (1) Neural Network

[0102] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0103]

[0104] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For xs , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to perform nonlinear transformation on the features obtained in the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0105] (2) Deep Neural Networks

[0106] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0107] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0108] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0109] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0110] (3) Convolutional Neural Network

[0111] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. Convolution kernels can be formalized as matrices of random size, and convolution kernels can obtain reasonable weights through learning during the training process of convolutional neural networks. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0112] (4) Loss Function

[0113] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process of optimization before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0114] (5) Back propagation algorithm

[0115] Neural networks can use the error back propagation (BP) algorithm to correct the values ​​of the parameters in the neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0116] The following introduces the system architecture provided by the embodiments of the present application.

[0117] like Figure 1 As shown, the embodiment of the present application provides a system architecture 100. Figure 1 In the embodiment of the present application, the data acquisition device 160 is used to collect training data. The training data in the embodiment of the present application may include a first training sample and / or a second training sample. The first training sample includes first human body three-dimensional point cloud data and the weight corresponding to the first human body three-dimensional point cloud data. The second training sample includes a first single-channel two-dimensional image and the human body key points corresponding to the first single-channel two-dimensional image. Among them, the weight corresponding to the first human body three-dimensional point cloud data may be the result of manual pre-annotation; the human body key points corresponding to the first single-channel two-dimensional image may be the result of manual pre-annotation.

[0118] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .

[0119] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input first human body three-dimensional point cloud data, and compares the output human body weight or human body weight range with the weight corresponding to the first human body three-dimensional point cloud data until the difference between the human body weight or human body weight range output by the training device 120 and the weight corresponding to the first human body three-dimensional point cloud data is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0120] Alternatively, the training device 120 processes the input first single-channel two-dimensional image, and compares the output human key points with the human key points corresponding to the first single-channel two-dimensional image, until the difference between the human key points output by the training device 120 and the human key points corresponding to the first single-channel two-dimensional image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0121] The above target model / rule 101 can be used to implement the human feature parameter detection method of the embodiment of the present application, that is, the single-channel two-dimensional image is input into the target model / rule 101 after relevant preprocessing to obtain the human key points, and then the human feature parameters are determined based on the human key points. The target model / rule 101 in the embodiment of the present application can specifically be an openpose network.

[0122] The above target model / rule 101 can be used to implement the weight detection method of the embodiment of the present application, that is, the human body weight can be obtained by inputting the three-dimensional point cloud data of the human body into the target model / rule 101 after relevant preprocessing. The target model / rule 101 in the embodiment of the present application can be specifically a pointnet network.

[0123] The above-mentioned target model / rule 101 can be used to implement the seat adjustment method of the embodiment of the present application, that is, the single-channel two-dimensional image and / or the three-dimensional point cloud data of the human body is input into the target model / rule 101 after relevant preprocessing to obtain the human body characteristic parameters and / or human body weight, and then the car seat parameters are adjusted based on the human body characteristic parameters and / or human body weight.

[0124] It should be noted that, in actual applications, the training data maintained in the database 130 may not all come from the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not train the target model / rule 101 entirely based on the training data maintained by the database 130, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0125] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 1 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: a two-dimensional image and a three-dimensional point cloud captured by a TOF camera.

[0126] Preprocessing module 113 and preprocessing module 114 are used to perform preprocessing according to the input data received by I / O interface 112 (such as two-dimensional images and three-dimensional point clouds captured by the TOF camera). In an embodiment of the present application, preprocessing module 113 and / or preprocessing module 114 can be used to preprocess the two-dimensional image and three-dimensional point cloud to obtain three-dimensional point cloud data of the human body.

[0127] In the embodiment of the present application, the preprocessing module 113 and the preprocessing module 114 may be omitted (or only one of the preprocessing modules may be provided), and the calculation module 111 may be directly used to process the input data.

[0128] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0129] Finally, the I / O interface 112 returns the processing results, such as the human characteristic parameters or human weight or seat parameters obtained above, to the client device 140, thereby providing them to the user or other devices to perform other operations, such as adjusting the car seat.

[0130] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0131] exist Figure 1 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0132] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0133] Since CNN is a very common neural network, Figure 2 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to an algorithm that updates the neural network model and performs multiple levels of learning at different levels of abstraction. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

[0134] exist Figure 2In FIG. 1 , a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230. Figure 3 The internal layer structure of CNN 200 is introduced in detail.

[0135] Convolutional layer / pooling layer 220:

[0136] Convolutional Layer:

[0137] like Figure 2 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0138] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0139] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0140] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0141] When the convolutional neural network 200 has multiple convolutional layers, the convolutional layers (for example, 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0142] Pooling layer:

[0143] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 2Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0144] Fully connected layer 230:

[0145] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the fully connected layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (such as Figure 3 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0146] After the multiple hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 3 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 3 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0147] It should be noted that Figure 2The convolutional neural network shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0148] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0149] Figure 3 A chip hardware structure provided in an embodiment of the present application includes a neural network processor 30. The chip can be arranged as follows: Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 2 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 3 The method in the embodiment of the present application can be executed by the neural network processor 30.

[0150] The neural network processor 30 can be any processor suitable for large-scale XOR operation processing, such as NPU, TPU, or GPU. Take NPU as an example: NPU can be mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of NPU is the operation circuit 303, which is controlled by the controller 304 to extract the matrix data in the memory (301 and 302) and perform multiplication and addition operations.

[0151] In some implementations, the operation circuit 303 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0152] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the weight data of the matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303. The operation circuit 303 takes the input data of the matrix A from the input memory 301, performs matrix operations based on the input data of the matrix A and the weight data of the matrix B, and stores the partial results or the final results of the matrix in the accumulator 308.

[0153] The vector calculation unit 307 can further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / fully connected layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0154] In some implementations, the vector calculation unit 307 stores the processed vectors in the unified memory 306. In some implementations, the vectors processed by the vector calculation unit 307 can be used as activation inputs of the operation circuit 303, for example, for use in subsequent layers in a neural network, such as Figure 2 As shown, if the current processing layer is hidden layer 1 (231), the vector processed by the vector calculation unit 307 can also be used in the calculation of hidden layer 2 (232).

[0155] The unified memory 306 is used to store input data and output data. The weight data is directly accessed by the direct memory access controller (DMAC) 305, which transfers the input data in the external memory to the input memory 301 and / or the unified memory 306, stores the weight data in the external memory into the weight memory 302, and stores the data in the unified memory 306 into the external memory.

[0156] The bus interface unit (BIU) 310 is used for the interaction between the DMAC and the instruction fetch buffer 309. The bus interface unit 310 is also used for the instruction fetch buffer 309 to obtain instructions from the external memory. The bus interface unit 301 is also used for the storage unit access controller 305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0157] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0158] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0159] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0160] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307. The operations of each layer of the neural network in the embodiment of the present application can be performed by the operation circuit 303 or the vector calculation unit 307. For example, in the embodiment of the present application, the operations of each layer in the image segmentation model can be performed by the operation circuit 303 or the vector calculation unit 307. For another example, in the embodiment of the present application, the operations of each layer in the weight assessment model can be performed by the operation circuit 303 or the vector calculation unit 307. For another example, in the embodiment of the present application, the key point detection model in the weight assessment model can be performed by the operation circuit 303 or the vector calculation unit 307.

[0161] The weight detection method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0162] Figure 4 This is a weight detection method 400 provided in an embodiment of the present application. Figure 4 The method shown can be used as Figure 1 The execution device 110 shown is executed. Optionally, the method 400 may be processed by a CPU, or by a CPU and a GPU, or may not use a GPU but use other processors suitable for neural network calculations, which is not limited in this application.

[0163] The method 400 includes steps S410 to S440. Steps S410 to S440 are described in detail below.

[0164] S410, acquiring a two-dimensional image captured by a time of flight (TOF) camera.

[0165] The layout position of the TOF camera can be set as needed to obtain the largest possible field of view so as to extract human body information over a large range.

[0166] The data collected by TOF camera is not limited by light and can be used during the day or at night. In addition, TOF camera has a large field of view, which is conducive to extracting human body information over a large range.

[0167] The two-dimensional image may be represented by two-dimensional data, for example, the two-dimensional image may be a grayscale image or an RGB image.

[0168] Optionally, the two-dimensional image may be a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth value corresponding to the pixel in the single-channel two-dimensional image, and the depth value may be obtained by depth data collected by a TOF camera.

[0169] The single-channel two-dimensional image is a two-dimensional image with contour information, and the depth value of the pixel is indicated by the gray value of the pixel. The single-channel two-dimensional image is a two-dimensional image with depth information.

[0170] Specifically, the depth value corresponding to each pixel can be converted into a grayscale value, that is, the grayscale value is used to represent the depth information, and a single-channel two-dimensional image is obtained.

[0171] Exemplarily, the gray value of the pixel is assigned in equal proportion according to the depth value of each pixel in the TOF camera, that is, the depth value depth corresponding to the pixel is assigned in equal proportion to the gray value gray_value corresponding to the pixel.

[0172] For example, the corresponding grayscale value of each pixel satisfies:

[0173]

[0174] In the above formula, the units of depth value and focal length are mm. For example, depth≥5000 in the above formula means that the depth value is greater than or equal to 5000 mm. The above formula can be understood as follows: if the depth value corresponding to a pixel is greater than or equal to 5 m, the grayscale value corresponding to the pixel is 0; if the depth value corresponding to the pixel is less than or equal to the focal length f of the camera, the grayscale value corresponding to the pixel is 255; if the depth value corresponding to the pixel is greater than the focal length f of the camera and less than 5 m, the grayscale value is assigned proportionally.

[0175] Furthermore, three-dimensional point cloud data corresponding to the two-dimensional image may be determined.

[0176] Here, point cloud refers to the combination of spatial coordinates of sampling points on the surface of an object.

[0177] Specifically, there is a mapping relationship between the imaging coordinates of the camera pixels and the coordinates of the three-dimensional point cloud. According to the mapping relationship and the depth data corresponding to the pixels, the three-dimensional point cloud data corresponding to the two-dimensional image can be determined.

[0178] For example, if the two-dimensional image is an RGB image, the three-dimensional point cloud data corresponding to the two-dimensional image can be determined based on the mapping relationship and the depth data corresponding to the pixels. If the two-dimensional image is a single-channel two-dimensional image, the three-dimensional point cloud data corresponding to the two-dimensional image can be determined based on the mapping relationship.

[0179] Exemplarily, the three-dimensional point cloud coordinates (X, Y, Z) can be calculated based on the focal length f of the camera, the imaging coordinates (x, y) of the camera pixels, and the depth data.

[0180] For example, Figure 5 As shown, the three-dimensional point cloud coordinates can be calculated based on the pinhole imaging model. The three-dimensional point cloud coordinates (X, Y, Z) satisfy:

[0181]

[0182] Among them, Z is the depth value of the point.

[0183] S420, detecting a human body region in the two-dimensional image.

[0184] Specifically, the two-dimensional image is segmented to obtain the human body region.

[0185] Optionally, the two-dimensional image is input into an image segmentation model to obtain a human body region. The image segmentation model may adopt a neural network model. For example, the image segmentation model may be U-net, V-net or deeplab-v3, etc.

[0186] Furthermore, the two-dimensional image may be a single-channel two-dimensional image in step S410. Detecting the human body region in a single-channel two-dimensional image can effectively utilize depth information, eliminate interference from other information such as color and brightness, and obtain a more accurate human body region.

[0187] For example, Figure 6 As shown, the single-channel two-dimensional image can be input into U-net to output the human body area.

[0188] The image segmentation model can be obtained by training based on the first training sample. The first training sample includes a first single-channel two-dimensional image and a human body region corresponding to the first single-channel two-dimensional image. The grayscale value corresponding to the pixel in the first single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the first single-channel two-dimensional image. Specifically, the image segmentation model can be trained by taking the first single-channel two-dimensional image as the input of the image segmentation model and taking the human body region corresponding to the first single-channel two-dimensional image as the target output of the image segmentation model.

[0189] The existing image segmentation model is based on RGB images. Using single-channel two-dimensional images as training samples for training can obtain a better image segmentation model and improve the accuracy of image segmentation.

[0190] The training method can be specifically Figure 1 The training device 120 shown in FIG. 1 is executed, and the first single-channel two-dimensional image can be as follows: Figure 1 The training data maintained in the database 130 shown, optionally, the step of obtaining the first single-channel two-dimensional image in the training process can be executed in the training device 120, or can be pre-executed by other functional modules before the training device 120. That is, the training data received or obtained from the database 130 is pre-processed to obtain the first single-channel two-dimensional image as the input of the training device 120, and the training device 120 executes the training process of the image segmentation model.

[0191] Optionally, the training process can be processed by the CPU, or by the CPU and GPU together, or other processors suitable for neural network calculations can be used without the GPU, and this application does not impose any restrictions.

[0192] S430, based on the depth information corresponding to the human body region in the two-dimensional image, convert the two-dimensional data of the human body region in the two-dimensional image into three-dimensional point cloud data of the human body, such as Fig.11 shown.

[0193] Exemplarily, the three-dimensional point cloud data of the human body corresponding to the human body region can be directly calculated according to the mapping relationship between the imaging coordinates in the camera pixels and the coordinates of the three-dimensional point cloud and the depth information corresponding to the human body region.

[0194] Alternatively, the three-dimensional point cloud data of the human body corresponding to the human body area may be extracted from the three-dimensional point cloud data corresponding to the two-dimensional image obtained in step S410 according to the mapping relationship between the imaging coordinates in the camera pixels and the coordinates of the three-dimensional point cloud.

[0195] Optionally, the two-dimensional image includes multiple frames of two-dimensional images, and the human body region in the multiple frames of two-dimensional images can be obtained in step S420.

[0196] In this case, step S430 includes: determining the three-dimensional point cloud data corresponding to the human body area in multiple frames of two-dimensional images according to the mapping relationship between the imaging coordinates in the camera pixels and the coordinates of the three-dimensional point cloud, and obtaining multiple frames of human body three-dimensional point cloud data; fusing the multiple frames of human body three-dimensional point cloud data to obtain human body three-dimensional point cloud data.

[0197] Alternatively, multiple frames of two-dimensional images correspond to multiple frames of three-dimensional point cloud data. For example, the multiple frames of three-dimensional point cloud data may be continuous frames of three-dimensional point cloud data.

[0198] In this case, step S430 includes extracting human body three-dimensional point cloud data corresponding to the human body area in multiple frames of three-dimensional point cloud to obtain multiple frames of human body three-dimensional point cloud data; and fusing the multiple frames of human body three-dimensional point cloud data to obtain human body three-dimensional point cloud data.

[0199] This can avoid the inability to obtain complete human body 3D point cloud data due to occlusion, posture, etc., and improve the accuracy of human body 3D point cloud data.

[0200] S440, determining the body weight according to the three-dimensional point cloud data of the body.

[0201] Specifically, the human body three-dimensional point cloud data can be input into a weight assessment model to obtain the human body weight. The human body weight can include a weight value or a weight range. The weight assessment model can be a neural network model. For example, the neural network model can include pointnet, etc.

[0202] For example, Figure 7 As shown, the 3D point cloud data of the human body is input into pointnet and the weight range is output.

[0203] Optionally, step S440 further includes: uniformly refining the three-dimensional point cloud data of the human body to obtain sparse three-dimensional point cloud data of the human body, and then determining the human body weight according to the sparse three-dimensional point cloud data of the human body.

[0204] For example, the farthest point sampling method can be used to evenly refine the human body 3D point cloud data, which can reduce the number of parameters, reduce the amount of calculation, and improve the processing speed.

[0205] Optionally, the weight assessment model can be obtained by training based on a second training sample, wherein the second training sample includes the first human body three-dimensional point cloud data and the weight corresponding to the first human body three-dimensional point cloud data. The weight corresponding to the first human body three-dimensional point cloud data can be regarded as a label corresponding to the first human body three-dimensional point cloud data. The weight assessment model is trained using the first human body point cloud data as the input of the weight assessment model and the corresponding human body weight as the target output of the weight assessment model.

[0206] For example, Figure 8 As shown, the first human body three-dimensional point cloud data and the corresponding labels are input into pointnet to train pointnet.

[0207] The weight assessment model is trained using the human body 3D point cloud data and the corresponding weight as training samples, so that the human body 3D point cloud data is regressed to the weight information to obtain the weight assessment model. In this way, the weight information can be directly determined through the human body 3D point cloud data without other calculations or conversions, such as no conversion of volume and density, etc., to avoid introducing other errors and improve the accuracy and processing speed of the weight assessment model.

[0208] The training method can be specifically Figure 1 The training device 120 shown in the figure is executed, and the three-dimensional point cloud data of the human body can be as follows Figure 1 The training data maintained in the database 130 shown, optionally, the uniform refinement of the three-dimensional point cloud data of the human body can be executed in the training device 120, or can be pre-executed by other functional modules before the training device 120, that is, the training data received or acquired from the database 130 is pre-processed, such as using the farthest point sampling method to obtain sparse three-dimensional point cloud data of the human body as the input of the training device 120, and the training device 120 performs training of the weight assessment model.

[0209] Optionally, the training method can be processed by a CPU, or by a CPU and a GPU together, or other processors suitable for neural network calculations can be used without a GPU, and this application does not impose any restrictions.

[0210] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain the human body area, the human body area is determined on a two-dimensional image, and then the three-dimensional point cloud data of the human body is obtained by mapping the two-dimensional image and the three-dimensional point cloud, which can improve the processing accuracy and processing speed, reduce energy consumption, and effectively reduce the heating of hardware. At the same time, based on the three-dimensional point cloud data of the human body, the human body weight is obtained, and more comprehensive and accurate human body morphology information can be obtained, which improves the accuracy of human body weight detection. At the same time, the use of the TOF visual system can further improve the detection accuracy, and relatively high accuracy can also be guaranteed at a long distance.

[0211] The human body characteristic parameter detection method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0212] Fig. 9 This is a human body characteristic parameter detection method 500 provided in an embodiment of the present application. Fig. 9 The method shown can be used as Figure 1 The execution device 110 shown is executed. Optionally, the method 500 may be processed by a CPU, or by a CPU and a GPU, or may not use a GPU but use other processors suitable for neural network calculations, which is not limited in this application.

[0213] The method 500 includes steps S510 to S540. Steps S510 to S540 are described in detail below.

[0214] S510, acquiring a two-dimensional image captured by a TOF camera.

[0215] The layout position of the TOF camera can be set as needed to obtain the largest possible field of view so as to extract human body information over a large range.

[0216] The data collected by TOF camera is not limited by light and can be used during the day or at night. In addition, TOF camera has a large field of view, which is conducive to extracting human body information over a large range.

[0217] The two-dimensional image may be a grayscale image or an RGB image.

[0218] Optionally, the two-dimensional image may be a single-channel two-dimensional image. The grayscale value corresponding to a pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image. The depth value corresponding to the pixel in the two-dimensional image may be obtained by depth data collected by a TOF camera.

[0219] The single-channel two-dimensional image is a two-dimensional image with contour information, and the depth information of the pixel is indicated by the gray value of the pixel. The single-channel two-dimensional image is a two-dimensional image with depth information.

[0220] Specifically, the depth value corresponding to each pixel can be converted into a grayscale value, that is, the grayscale value is used to represent the depth information, and a single-channel two-dimensional image is obtained.

[0221] Exemplarily, the gray value may be assigned in equal proportion, that is, the depth value depth corresponding to the pixel and the gray value gray_value corresponding to the pixel are assigned in equal proportion.

[0222] For example, the corresponding grayscale value of each pixel satisfies:

[0223]

[0224] In the above formula, the units of depth value and focal length are mm. For example, depth≥5000 in the above formula means that the depth value is greater than or equal to 5000 mm. The above formula can be understood as follows: if the depth value corresponding to a pixel is greater than or equal to 5 m, the grayscale value corresponding to the pixel is 0; if the depth value corresponding to the pixel is less than or equal to the focal length f, the grayscale value corresponding to the pixel is 255; if the depth value corresponding to the pixel is greater than the focal length f and less than 5 m, the grayscale value is assigned proportionally.

[0225] Furthermore, three-dimensional point cloud data corresponding to the two-dimensional image may be determined.

[0226] Specifically, there is a mapping relationship between the imaging coordinates in the camera pixels and the coordinates of the three-dimensional point cloud. According to the mapping relationship and the depth data, the three-dimensional point cloud data corresponding to the two-dimensional image can be determined.

[0227] For example, if the two-dimensional image is an RGB image, the three-dimensional point cloud data corresponding to the two-dimensional image can be determined based on the mapping relationship and the depth data corresponding to the pixels. If the two-dimensional image is a single-channel two-dimensional image, the three-dimensional point cloud data corresponding to the two-dimensional image can be determined based on the mapping relationship.

[0228] Exemplarily, the three-dimensional point cloud coordinates (X, Y, Z) can be calculated based on the focal length f of the camera, the imaging coordinates (x, y) of the camera pixels, and the depth data.

[0229] For example, Figure 5 As shown, the three-dimensional point cloud coordinates can be calculated based on the pinhole imaging model. The three-dimensional point cloud coordinates (X, Y, Z) satisfy:

[0230]

[0231] Among them, Z is the depth value of the point.

[0232] S520, detecting key points of a human body in the two-dimensional image.

[0233] The number and position of the key points of the human body can be set as needed. For example, the key points of the human body may include: elbow joints, shoulder joints, top of the head, ankles or wrists.

[0234] Specifically, the two-dimensional image is input into a key point detection model to obtain the key points of the human body. The key point detection model may be a neural network model. For example, the neural network model may include openpose.

[0235] Furthermore, the two-dimensional image may be a single-channel two-dimensional image in step S510. Using a single-channel two-dimensional image to detect key points of a human body can effectively utilize depth information, eliminate interference from other information such as color and brightness, and obtain more accurate key points of a human body.

[0236] For example, Fig.10 As shown, the single-channel two-dimensional image can be input into the mobilenet network to extract feature information, and then the feature information can be input into the openpose network to detect the key points of the human body and output the two-dimensional coordinates of the key points of the human body.

[0237] Optionally, the key point detection model is trained based on a third training sample. The third training sample includes a second single-channel two-dimensional image and human key points corresponding to the second single-channel two-dimensional image. The grayscale value corresponding to the pixel in the second single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the second single-channel two-dimensional image.

[0238] Existing key point detection models are usually based on RGB images. In the embodiments of the present application, single-channel two-dimensional images are used as training samples for training, which can obtain a better key point detection model and improve the accuracy of key point detection.

[0239] Alternatively, if the two-dimensional image is an RGB image, the RGB image is input into a key point detection model to obtain human key points. The key point detection model is obtained by training based on a fourth training sample. The fourth training sample includes the first RGB image and the human key points corresponding to the first RGB image.

[0240] The training method of the above key point model can be specifically as follows: Figure 1 The training device 120 shown in FIG. 1 is executed, and the second single-channel two-dimensional image can be as follows: Figure 1 The training data maintained in the database 130 shown, optionally, the step of obtaining the second single-channel two-dimensional image in the training process can be performed in the training device 120, or can be pre-performed by other functional modules before the training device 120. That is, the training data received or obtained from the database 130 is pre-processed to obtain the second single-channel two-dimensional image as the input of the training device 120, and the training device 120 performs the training process of the key point detection model.

[0241] Optionally, the training process can be processed by the CPU, or by the CPU and GPU together, or other processors suitable for neural network calculations can be used without the GPU, and this application does not impose any restrictions.

[0242] S530, based on the depth information corresponding to the human body key points in the two-dimensional image, convert the two-dimensional data of the human body key points in the two-dimensional image into three-dimensional point cloud data, and obtain the three-dimensional point cloud data corresponding to the human body key points, that is, determine the three-dimensional point cloud coordinates of the human body key points.

[0243] Exemplarily, the three-dimensional point cloud data corresponding to the key points of the human body can be directly determined based on the mapping relationship between the imaging coordinates of the camera pixels and the coordinates of the three-dimensional point cloud and the depth information corresponding to the key points of the human body.

[0244] Alternatively, the three-dimensional point cloud data corresponding to the key points of the human body may be extracted from the three-dimensional point cloud data corresponding to the two-dimensional image obtained in step S510 according to the mapping relationship between the imaging coordinates of the camera pixels and the coordinates of the three-dimensional point cloud.

[0245] S540, determining human body characteristic parameters according to the three-dimensional point cloud data corresponding to the key points of the human body. The human body characteristic parameters include: height, arm length, leg length or shoulder width, etc.

[0246] Fig.11 A schematic diagram of human body point cloud data is provided. Specifically, Fig.11 As shown, the distance between the key points of the human body can be calculated based on the three-dimensional point cloud data of the key points of the human body, and the human body characteristic parameters can be determined based on the distance. Fig.11 The three-dimensional point cloud data corresponding to some key points of the human body are shown to calculate the arm length and leg length.

[0247] Fig.12 It should be understood that for ease of description, the key points in a single-channel two-dimensional image are shown. Fig.12 The calculation process of human body characteristic parameters is illustrated by taking the human body key points of the two-dimensional image in as an example. However, in the actual calculation process, the human body characteristic parameters are determined based on the three-dimensional point cloud data corresponding to the human body key points. For example, Fig.12 8 key points are shown in FIG. The height of a person can be determined based on the distance between point 1 and point 8; the arm length can be determined based on the distance between point 3 and point 4; the leg length can be determined based on the distance between point 6 and point 8; and the shoulder width can be determined based on the distance between point 2 and point 3. It should be understood that Fig.12 The eight key points in the figure are only examples. The number and position of the key points actually used can be determined as needed. For example, when determining the arm length, the key point at the elbow can be added, and the arm length information can be determined based on the distance between point 3 and the key point at the elbow and the distance between point 4 and the key point at the elbow. In this way, the arm length information can be obtained even if the arm is not straight. The embodiment of the present application does not limit the number and position of the key points.

[0248] Optionally, the two-dimensional image in step S510 may include multiple frames of two-dimensional images.

[0249] Optionally, steps S520 to S540 are performed on one frame of the multiple two-dimensional images to determine the human body feature parameters.

[0250] Alternatively, steps S520 to S540 are performed on multiple frames of two-dimensional images to obtain multiple groups of human body feature parameters, and the final human body feature parameters are determined based on the multiple groups of human body feature parameters. For example, steps S520 to S540 are performed on n frames of two-dimensional images to obtain n groups of heights, and the average value of the n groups of heights is calculated, and the average value is used as the final height.

[0251] By optimizing the human feature parameter results determined by multiple frames of two-dimensional images, it is possible to avoid the inability to obtain the correct key point position due to occlusion, posture, etc., thereby improving the accuracy of human feature parameter detection.

[0252] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain key points, key point detection on a two-dimensional image can improve processing accuracy and processing speed, reduce energy consumption, and effectively reduce hardware heating. At the same time, determining human feature parameters based on three-dimensional point cloud data of human key points improves the accuracy of human feature parameter detection. At the same time, the use of a TOF visual system can further improve the detection accuracy of human feature parameters, and relatively high accuracy can also be guaranteed at a long distance.

[0253] The seat adjustment method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0254] Method 400 and / or method 500 may be applied to seat adjustment. Specifically, the parameters of the car seat are adjusted according to the body weight obtained by method 400 and / or the body characteristic parameters obtained by method 500.

[0255] Fig.13 A seat adjustment method 600 of an embodiment of the present application is shown. The specific implementation of method 600 can refer to the aforementioned method 400. In order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing method 600. Method 600 includes steps S610 to S650, and steps S610 to S650 are described in detail below.

[0256] S610, acquiring a two-dimensional image captured by a TOF camera.

[0257] The TOF camera can be placed outside the vehicle, so that the passenger's information can be obtained before the passenger gets on the vehicle, and the seat parameters can be adjusted in advance to improve the user experience.

[0258] Alternatively, if Fig.14 As shown in the figure, the TOF camera can be placed at the junction of the B-pillar and the roof of the car. The inclination angle of this position is large, and a larger field of view can be obtained, as shown in the dotted line range in the figure. This can cover most of the angles of getting on the car and extract more comprehensive information about the passengers.

[0259] Optionally, step S610 further includes determining three-dimensional point cloud data corresponding to the two-dimensional image according to a mapping relationship between imaging coordinates in camera pixels and coordinates of the three-dimensional point cloud and depth data.

[0260] Exemplarily, the three-dimensional point cloud coordinates (X, Y, Z) can be calculated based on the focal length f of the camera, the imaging coordinates (x, y) of the camera pixels, and the depth data.

[0261] Optionally, the two-dimensional image is a grayscale image or an RGB image.

[0262] Or the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth value corresponding to the pixel in the single-channel two-dimensional image.

[0263] Specifically, the depth value corresponding to each pixel can be converted into a grayscale value, that is, the grayscale value is used to represent the depth information, and a single-channel two-dimensional image is obtained.

[0264] Exemplarily, the grayscale value of the pixel is assigned in proportion according to the depth value of each pixel in the TOF camera. For example, if the depth value depth corresponding to pixel A is greater than or equal to 5m, the grayscale value corresponding to pixel A is 0; if the depth value depth corresponding to pixel A is less than or equal to the focal length f of the TOF camera, the grayscale value corresponding to pixel A is 255; if the depth value depth corresponding to pixel A is greater than the focal length of the TOF camera and less than 5m, the grayscale value corresponding to the pixel A is assigned in proportion.

[0265] Optionally, step S610 includes: performing face recognition on the two-dimensional image, and if the user in the two-dimensional image exists in the database, adjusting the parameters of the car seat according to the parameters in the database. If the user in the two-dimensional image does not exist in the database, executing step S620.

[0266] S620: Detect a human body region in the two-dimensional image.

[0267] Optionally, the two-dimensional image is input into an image segmentation model to obtain a human body region. The image segmentation model may adopt a neural network model. For example, the image segmentation model may be U-net, V-net or deeplab-v3, etc.

[0268] Alternatively, the single-channel two-dimensional image in step S611 can be input into the image segmentation model to obtain the human body region. Using the single-channel two-dimensional image for image segmentation can effectively utilize depth information, eliminate interference from other information such as color and brightness, and obtain a more accurate human body region.

[0269] S630: Convert the two-dimensional data of the human body region in the two-dimensional image into three-dimensional point cloud data of the human body based on the depth information corresponding to the human body region in the two-dimensional image.

[0270] If three-dimensional point cloud data corresponding to the two-dimensional image is obtained in step S610, three-dimensional point cloud data of the human body corresponding to the human body area can be extracted from the three-dimensional point cloud data corresponding to the two-dimensional image according to the mapping relationship between the imaging coordinates in the camera pixels and the coordinates of the three-dimensional point cloud.

[0271] Optionally, the two-dimensional image includes multiple frames of two-dimensional images, and the human body region in the multiple frames of two-dimensional images can be obtained in step S620.

[0272] The multiple frames of two-dimensional images correspond to multiple frames of three-dimensional point cloud data. For example, the multiple frames of three-dimensional point cloud data may be continuous frames of three-dimensional point cloud data.

[0273] In this case, step S630 includes extracting human body three-dimensional point cloud data corresponding to the human body area in multiple frames of three-dimensional point cloud to obtain multiple frames of human body three-dimensional point cloud data; and fusing multiple frames of human body three-dimensional point cloud data to obtain human body three-dimensional point cloud data.

[0274] This can avoid the inability to obtain complete human body 3D point cloud data due to occlusion, posture, etc., and improve the accuracy of human body 3D point cloud data.

[0275] S640, determining the body weight according to the three-dimensional point cloud data of the body.

[0276] Specifically, the human body three-dimensional point cloud data can be input into a weight assessment model to obtain the human body weight. The human body weight can include a weight value or a weight range. The weight assessment model can be a neural network model. For example, the neural network model can include pointnet, etc.

[0277] Optionally, step S640 further includes: uniformly refining the three-dimensional point cloud data of the human body to obtain sparse three-dimensional point cloud data of the human body, and then determining the human body weight according to the sparse three-dimensional point cloud data of the human body.

[0278] For example, the farthest point sampling method can be used to evenly refine the human body 3D point cloud data, which can reduce the number of parameters, reduce the amount of calculation, and improve the processing speed.

[0279] S650, adjusts car seat parameters according to human body weight.

[0280] Car seat parameters include the position of the car or the angle of the seat back, etc.

[0281] Exemplarily, the parameters of the seat may be obtained according to the body weight of the human body by matching the correspondence between the body weight of the human body and the parameters of the seat in the database.

[0282] Alternatively, the body weight can be input into the seat adjustment model to obtain the parameters of the seat. The seat adjustment model can be a neural network model. The seat adjustment model can be trained based on a body weight sample and the parameters of the seat corresponding to the body weight sample. The parameters of the seat can be the result of manual annotation. Specifically, the body weight sample is used as the input of the seat adjustment model, and the parameters of the seat are used as the target output value of the seat adjustment model for training to obtain a trained seat adjustment model.

[0283] Optionally, step S650 may include: adjusting the parameters of the car seat according to the human body characteristic parameters and the human body weight. The human body characteristic parameters may be obtained by method 500, such as Fig.16 shown.

[0284] Exemplarily, the parameters of the seat can be obtained by matching the corresponding relationship between the human body characteristic parameters and the human body weight and the parameters of the seat in the database according to the human body characteristic parameters and the human body weight.

[0285] Alternatively, the human body characteristic parameters and human body weight may be input into the seat adjustment model to obtain the parameters of the seat. The seat adjustment model may be a neural network model. The seat adjustment model may be trained based on human body characteristic parameter samples, human body weight samples and corresponding seat parameters. The seat parameters may be the result of manual annotation. Specifically, the human body characteristic parameter samples and body weight samples are used as inputs to the seat adjustment model, and the seat parameters are used as target output values ​​of the seat adjustment model for training to obtain a trained seat adjustment model.

[0286] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain the human body area, determining the human body area on a two-dimensional image and then obtaining the three-dimensional point cloud data of the human body can improve the processing accuracy and processing speed, reduce energy consumption, and effectively reduce hardware heating. At the same time, obtaining the human body weight based on the three-dimensional point cloud data of the human body can obtain more comprehensive and accurate human body morphological information, improve the accuracy of human body weight detection, better adjust the parameters of the car seat, and improve the user experience. At the same time, the use of the TOF visual system can further improve the detection accuracy, and can also ensure relatively high accuracy at a long distance.

[0287] Fig.15 A seat adjustment method 700 of an embodiment of the present application is shown. The specific implementation of method 700 can refer to the aforementioned method 500. In order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing method 700. Method 700 includes steps S710 to S750, and steps S710 to S750 are described in detail below.

[0288] S710, acquiring a two-dimensional image captured by a TOF camera.

[0289] The TOF camera can be placed outside the vehicle, so that the passenger's information can be obtained before the passenger gets on the vehicle, and the seat parameters can be adjusted in advance to improve the user experience.

[0290] For example, Fig.14 As shown in the figure, the TOF camera can be placed at the junction of the B-pillar and the roof of the car. The inclination angle of this position is large, and a larger field of view can be obtained, as shown in the dotted line range in the figure. This can cover most of the angles of getting on the car and extract more comprehensive information about the passengers.

[0291] Optionally, step S710 further includes determining three-dimensional point cloud data corresponding to the two-dimensional image according to a mapping relationship between imaging coordinates in camera pixels and coordinates of the three-dimensional point cloud and depth data.

[0292] Exemplarily, the three-dimensional point cloud coordinates (X, Y, Z) can be calculated based on the focal length f of the camera, the imaging coordinates (x, y) of the camera pixels, and the depth data.

[0293] Optionally, the two-dimensional image is a grayscale image or an RGB image.

[0294] Or the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth value corresponding to the pixel in the single-channel two-dimensional image.

[0295] Specifically, the depth value corresponding to each pixel can be converted into a grayscale value, that is, the grayscale value is used to represent the depth information, and a single-channel two-dimensional image is obtained.

[0296] Exemplarily, the grayscale value of the pixel is assigned in proportion according to the depth value of each pixel in the TOF camera. For example, if the depth value depth corresponding to pixel A is greater than or equal to 5m, the grayscale value corresponding to pixel A is 0; if the depth value depth corresponding to pixel A is less than or equal to the focal length f of the TOF camera, the grayscale value corresponding to pixel A is 255; if the depth value depth corresponding to pixel A is greater than the focal length of the TOF camera and less than 5m, the grayscale value corresponding to the pixel A is assigned in proportion.

[0297] Optionally, step S710 includes: performing face recognition on the two-dimensional image, and if the user in the two-dimensional image exists in the database, adjusting the parameters of the car seat according to the parameters in the database. If the user in the two-dimensional image does not exist in the database, executing step S720.

[0298] S720, detecting key points of a human body in the two-dimensional image.

[0299] Specifically, the two-dimensional image is input into a key point detection model to obtain the key points of the human body. The key point detection model may be a neural network model. For example, the neural network model may include openpose.

[0300] Alternatively, the single-channel two-dimensional image obtained in step S710 can be input into the key point detection model. Using a single-channel two-dimensional image to detect human key points can effectively use depth information, eliminate interference from other information such as color and brightness, and obtain more accurate human key points.

[0301] S730, converting the two-dimensional data of the human body key points in the two-dimensional image into three-dimensional point cloud data based on the depth information corresponding to the human body key points in the two-dimensional image, to obtain the three-dimensional point cloud data corresponding to the human body key points.

[0302] If three-dimensional point cloud data corresponding to the two-dimensional image is obtained in step S710, three-dimensional point cloud data corresponding to the key points of the human body can be extracted from the three-dimensional point cloud data corresponding to the two-dimensional image according to the mapping relationship between the imaging coordinates in the camera pixels and the coordinates of the three-dimensional point cloud.

[0303] S740, determining human body characteristic parameters according to the three-dimensional point cloud data corresponding to the key points of the human body. The human body characteristic parameters include: height, arm length, leg length or shoulder width, etc.

[0304] Specifically, the distance between key points of the human body can be calculated based on the three-dimensional point cloud data of the key points of the human body, and the human body characteristic parameters can be determined based on the distance.

[0305] S750, adjusting the parameters of the car seat according to the human body characteristic parameters. The car seat parameters include the position of the car or the angle of the seat back, etc.

[0306] Exemplarily, the parameters of the seat may be obtained by matching the corresponding relationship between the human body characteristic parameters and the parameters of the seat in the database according to the human body characteristic parameters.

[0307] Alternatively, the human body characteristic parameters may be input into the seat adjustment model to obtain the parameters of the seat. The seat adjustment model may be a neural network model. The seat adjustment model may be obtained by training based on human body characteristic parameter samples and the parameters of the seat corresponding to the human body characteristic parameter samples. The parameters of the seat may be the result of manual annotation. Specifically, the human body characteristic parameter samples are used as the input of the seat adjustment model, and the parameters of the seat are used as the target output values ​​of the seat adjustment model for training to obtain a trained seat adjustment model.

[0308] Optionally, step S750 may include: adjusting the parameters of the car seat according to the human body characteristic parameters and the human body weight. The human body weight may be obtained by method 400, such as Fig.16 shown.

[0309] Exemplarily, the parameters of the seat can be obtained by matching the corresponding relationship between the human body characteristic parameters and the human body weight and the parameters of the seat in the database according to the human body characteristic parameters and the human body weight.

[0310] Alternatively, the human body characteristic parameters and human body weight may be input into the seat adjustment model to obtain the parameters of the seat. The seat adjustment model may be a neural network model. The seat adjustment model may be trained based on human body characteristic parameter samples, human body weight samples and corresponding seat parameters. The seat parameters may be the result of manual annotation. Specifically, the human body characteristic parameter samples and body weight samples are used as inputs to the seat adjustment model, and the seat parameters are used as target output values ​​of the seat adjustment model for training to obtain a trained seat adjustment model.

[0311] In the embodiment of the present application, compared with directly processing in three-dimensional space to obtain key points, key point detection on a two-dimensional image can improve processing accuracy and processing speed, reduce energy consumption, and effectively reduce hardware heating. At the same time, the human body characteristic parameters are determined based on the three-dimensional point cloud data of the key points of the human body, which improves the accuracy of human body characteristic parameter detection, can better adjust the parameters of the car seat, and improve the user experience. At the same time, the use of the TOF visual system can further improve the detection accuracy, and can also ensure relatively high accuracy at a long distance.

[0312] Fig.16 This is a schematic flow chart of another seat adjustment method in an embodiment of the present application. The specific implementation method can refer to the aforementioned method 400 and method 500. In order to avoid unnecessary repetition, repeated descriptions are appropriately omitted in the following introduction. Fig.16 As shown, according to the mapping relationship between the imaging coordinates of the camera pixels and the three-dimensional point cloud, the three-dimensional point cloud corresponding to the two-dimensional image is obtained. The two-dimensional image collected by the TOF camera is respectively subjected to key point detection and image segmentation to obtain the key points of the human body and the human body area. Further, according to the mapping relationship between the imaging coordinates of the camera pixels and the three-dimensional point cloud, the three-dimensional point cloud data of the human body and the three-dimensional point cloud corresponding to the key points of the human body are determined. Then, according to the key points of the human body, the characteristic parameters of the human body, such as height, leg length or shoulder width, are determined. And the three-dimensional point cloud of the human body is input into a weight assessment model, such as a neural network model, to obtain the weight of the human body. The parameters of the car seat are adjusted according to the weight of the human body and the characteristic parameters of the human body. Fig.17 It is a schematic block diagram of a weight detection device according to an embodiment of the present application. Fig.17 The weight detection device 800 shown includes an acquisition unit 810 and a processing unit 820 .

[0313] The acquisition unit 810 and the processing unit 820 may be used to execute the weight detection method of the embodiment of the present application. Specifically, the acquisition unit 810 may execute the above step S410, and the processing unit 820 may execute the above steps S420 to S430.

[0314] The acquisition unit 810 is used to acquire a two-dimensional image captured by a time-of-flight TOF camera. The processing unit 820 is used to: detect a human body region in the two-dimensional image; convert the two-dimensional data of the human body region in the two-dimensional image into three-dimensional point cloud data of the human body based on the depth information corresponding to the human body region in the two-dimensional image; and determine the weight of the human body according to the three-dimensional point cloud data of the human body.

[0315] Optionally, as an embodiment, the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image.

[0316] Optionally, as an embodiment, the processing unit 820 is specifically used to: input the single-channel two-dimensional image into an image segmentation model to obtain a human body region, wherein the image segmentation model is trained based on a first training sample, the first training sample includes a first single-channel two-dimensional image and a human body region corresponding to the first single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the first single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the first single-channel two-dimensional image.

[0317] Optionally, as an embodiment, the processing unit 820 is specifically used to: input the human body three-dimensional point cloud data into a weight assessment model to obtain the human body weight, wherein the weight assessment model is trained based on a second training sample, and the second training sample includes first human body three-dimensional point cloud data and the weight corresponding to the first human body three-dimensional point cloud data.

[0318] Optionally, as an embodiment, the two-dimensional image includes multiple frames of two-dimensional images, and the processing unit 820 is specifically used to: convert the two-dimensional data of the human body area in the multiple frames of two-dimensional images into human body three-dimensional point cloud data based on the depth information corresponding to the human body area in the multiple frames of two-dimensional images to obtain multiple frames of human body three-dimensional point cloud data; and fuse the multiple frames of human body three-dimensional point cloud data to obtain human body three-dimensional point cloud data.

[0319] Optionally, as an embodiment, the processing unit 820 is specifically used to: uniformly refine the three-dimensional point cloud data of the human body to obtain sparse three-dimensional point cloud data of the human body; and determine the weight of the human body according to the sparse three-dimensional point cloud data of the human body.

[0320] Fig.18It is a schematic block diagram of a human body characteristic parameter detection device according to an embodiment of the present application. Fig.18 The human body characteristic parameter detection device 900 shown includes an acquisition unit 910 and a processing unit 920 .

[0321] The acquisition unit 910 and the processing unit 920 can be used to execute the human feature parameter detection method of the embodiment of the present application. Specifically, the acquisition unit 910 can execute the above step S510, and the processing unit 920 can execute the above steps S520 to S540.

[0322] The acquisition unit 910 is used to acquire a two-dimensional image captured by a time-of-flight TOF camera. The processing unit 920 is used to: detect human key points in the two-dimensional image; convert the two-dimensional data of the human key points in the two-dimensional image into three-dimensional point cloud data based on the depth information corresponding to the human key points in the two-dimensional image, and obtain the three-dimensional point cloud data corresponding to the human key points; determine human feature parameters according to the three-dimensional point cloud data corresponding to the human key points, wherein the human feature parameters include: height, arm length, leg length or shoulder width.

[0323] Optionally, as an embodiment, the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image.

[0324] Optionally, as an embodiment, the processing unit 920 is specifically used to: input the single-channel two-dimensional image into a key point detection model to obtain human key points, wherein the key point detection model is trained based on a third training sample, the third training sample includes a second single-channel two-dimensional image and human key points corresponding to the second single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the second single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the second channel two-dimensional image.

[0325] Fig.19 It is a schematic block diagram of a seat adjustment device according to an embodiment of the present application. Fig.19 The seat adjustment device 1000 shown includes an acquisition unit 1010 and a processing unit 1020 .

[0326] The acquisition unit 1010 and the processing unit 1020 can be used to execute the seat adjustment method of the embodiment of the present application. Specifically, the acquisition unit 1010 can execute the above-mentioned step S610 or step S710, and the processing unit 1020 can execute the above-mentioned steps S620 to S650 or steps S720 to S750.

[0327] The acquisition unit 1010 is used to acquire a two-dimensional image captured by a TOF camera. The processing unit 1020 is used to: detect key points of the human body in the two-dimensional image; convert the two-dimensional data of the key points of the human body in the two-dimensional image into three-dimensional point cloud data based on the depth information corresponding to the key points of the human body in the two-dimensional image, and obtain the three-dimensional point cloud data corresponding to the key points of the human body; determine human feature parameters according to the three-dimensional point cloud data corresponding to the key points of the human body, wherein the human feature parameters include: one or more of height, arm length, leg length or shoulder width; and adjust the parameters of the car seat according to the human feature parameters.

[0328] Optionally, as an embodiment, the processing unit 1020 is also used to: detect a human body area in the two-dimensional image; convert the two-dimensional data of the human body area in the two-dimensional image into three-dimensional point cloud data of the human body based on depth information corresponding to the human body area in the two-dimensional image; determine the human body weight according to the three-dimensional point cloud data of the human body; and adjust the car seat parameters according to the human body weight and the human body characteristic parameters.

[0329] Optionally, as an embodiment, the TOF camera is located at the junction of the B-pillar and the roof of the car.

[0330] Optionally, as an embodiment, the two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image.

[0331] It should be noted that the above-mentioned apparatus 800, apparatus 900 and apparatus 1000 are embodied in the form of functional units. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0332] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0333] Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0334] Fig. 20 It is a schematic diagram of the hardware structure of the weight detection device provided in the embodiment of the present application. Fig. 20 The weight detection device 1100 shown (the device 1100 may be a computer device) includes a memory 1101, a processor 1102, a communication interface 1103 and a bus 1104. The memory 1101, the processor 1102 and the communication interface 1103 are connected to each other through the bus 1104.

[0335] The memory 1101 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1101 may store a program. When the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute the various steps of the weight detection method of the embodiment of the present application.

[0336] Processor 1102 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the weight detection device of the embodiment of the present application, or to execute the weight detection method of the method embodiment of the present application.

[0337] The processor 1102 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the weight detection method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1102. The above-mentioned processor 1102 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1101, and the processor 1102 reads the information in the memory 1101, and combines its hardware to complete the functions required to be performed by the units included in the weight detection device of the embodiment of the present application, or executes the weight detection method of the method embodiment of the present application.

[0338] The communication interface 1103 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1100 and other devices or a communication network. For example, a two-dimensional image can be acquired through the communication interface 1103 .

[0339] The bus 1104 may include a path for transmitting information between various components of the device 1100 (eg, the memory 1101 , the processor 1102 , and the communication interface 1103 ).

[0340] It should be understood that the acquisition unit 810 in the weight detection device 800 is equivalent to the communication interface 1103 in the weight detection device 1100 ; and the processing unit 820 in the weight detection device 800 may be equivalent to the processor 1102 .

[0341] Fig.21 It is a schematic diagram of the hardware structure of the human body characteristic parameter detection device provided in an embodiment of the present application. Fig.21 The human body characteristic parameter device 1200 shown (the device 1200 may be a computer device) includes a memory 1201, a processor 1202, a communication interface 1203 and a bus 1204. The memory 1201, the processor 1202 and the communication interface 1203 are connected to each other through the bus 1204.

[0342] The memory 1201 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1201 may store a program. When the program stored in the memory 1201 is executed by the processor 1202, the processor 1202 and the communication interface 1203 are used to execute the various steps of the human body feature parameter method of the embodiment of the present application.

[0343] Processor 1202 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the human feature parameter device of the embodiment of the present application, or to execute the human feature parameter method of the method embodiment of the present application.

[0344] The processor 1202 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the human body characteristic parameter method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1202. The above-mentioned processor 1202 may also be a general-purpose processor, a digital signal processor (digitalsignal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmablegate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1201, and the processor 1202 reads the information in the memory 1201, and combines its hardware to complete the functions required to be performed by the units included in the human feature parameter device of the embodiment of the present application, or executes the human feature parameter method of the method embodiment of the present application.

[0345] The communication interface 1203 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1200 and other devices or a communication network. For example, a two-dimensional image can be acquired through the communication interface 1203 .

[0346] The bus 1204 may include a path for transmitting information between various components of the device 1200 (eg, the memory 1201 , the processor 1202 , and the communication interface 1203 ).

[0347] It should be understood that the acquisition unit 910 in the human body feature parameter device 900 is equivalent to the communication interface 1203 in the human body feature parameter device 1200 ; and the processing unit 920 in the human body feature parameter device 900 may be equivalent to the processor 1202 .

[0348] Fig. 22 It is a schematic diagram of the hardware structure of the seat adjustment device provided in an embodiment of the present application. Fig. 20 The seat adjustment device 1300 shown (the device 1300 may be a computer device) includes a memory 1301, a processor 1302, a communication interface 1303 and a bus 1304. The memory 1301, the processor 1302 and the communication interface 1303 are connected to each other through the bus 1304.

[0349] The memory 1301 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1301 may store a program. When the program stored in the memory 1301 is executed by the processor 1302, the processor 1302 and the communication interface 1303 are used to execute the various steps of the seat adjustment method of the embodiment of the present application.

[0350] Processor 1302 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the seat adjustment device of the embodiment of the present application, or to execute the seat adjustment method of the method embodiment of the present application.

[0351] The processor 1302 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the seat adjustment method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1302. The above-mentioned processor 1302 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1301, and the processor 1302 reads the information in the memory 1301, and combines its hardware to complete the functions required to be performed by the units included in the seat adjustment device of the embodiment of the present application, or executes the seat adjustment method of the method embodiment of the present application.

[0352] The communication interface 1303 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1300 and other devices or a communication network. For example, a two-dimensional image can be acquired through the communication interface 1303 .

[0353] The bus 1304 may include a path for transmitting information between various components of the device 1300 (eg, the memory 1301 , the processor 1302 , and the communication interface 1303 ).

[0354] It should be understood that the acquisition unit 1010 in the seat adjustment device 1000 is equivalent to the communication interface 1303 in the seat adjustment device 1300 ; and the processing unit 1020 in the seat adjustment device 1000 may be equivalent to the processor 1302 .

[0355] It should be noted that although Fig. 20 , Fig.21 , Fig. 22The device shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 1100, the device 1200, and the device 1300 also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 1100, the device 1200, and the device 1300 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 1100, the device 1200, and the device 1300 may also only include the devices necessary to implement the embodiments of the present application, and do not necessarily include Fig. 20 , Fig.21 , Fig. 22 All devices shown in .

[0356] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0357] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0358] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0359] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0360] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0361] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0362] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0363] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0364] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0365] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0366] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0367] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0368] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0369] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting body weight, It is characterized in that include: Get the two-dimensional image captured by the time-of-flight TOF camera, where: The two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image; Detecting a human body region in the single-channel two-dimensional image based on an image segmentation model, wherein the image segmentation model is obtained by training based on a first training sample, wherein the first training sample includes a first single-channel two-dimensional image and a human body region corresponding to the first single-channel two-dimensional image, and a grayscale value corresponding to a pixel in the first single-channel two-dimensional image is used to indicate depth information corresponding to the pixel in the first single-channel two-dimensional image; Converting the two-dimensional data of the human body region in the single-channel two-dimensional image into three-dimensional point cloud data of the human body based on the depth information corresponding to the human body region in the single-channel two-dimensional image; The three-dimensional point cloud data of the human body is input into a weight assessment model to obtain the human body weight.

2. The method according to claim 1, It is characterized in that The weight assessment model is obtained by training based on a second training sample, where the second training sample includes first human body three-dimensional point cloud data and a weight corresponding to the first human body three-dimensional point cloud data.

3. The method according to claim 2, It is characterized in that The single-channel two-dimensional image includes multiple frames of single-channel two-dimensional images, and The converting the two-dimensional data of the human body region in the single-channel two-dimensional image into three-dimensional point cloud data of the human body based on the depth information corresponding to the human body region in the single-channel two-dimensional image comprises: Based on the depth information corresponding to the human body region in the multiple frames of single-channel two-dimensional images, converting the two-dimensional data of the human body region in the multiple frames of single-channel two-dimensional images into human body three-dimensional point cloud data to obtain multiple frames of human body three-dimensional point cloud data; The multiple frames of human body three-dimensional point cloud data are fused to obtain fused human body three-dimensional point cloud data.

4. The method according to any one of claims 1 to 3, It is characterized in that Determining the body weight according to the three-dimensional point cloud data of the body includes: Uniformly refining the three-dimensional point cloud data of the human body to obtain sparse three-dimensional point cloud data of the human body; The body weight is determined according to the sparse three-dimensional point cloud data of the body.

5. A method for detecting human body characteristic parameters, It is characterized in that include: Get the two-dimensional image captured by the time-of-flight TOF camera, where: The two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image; Detecting human key points in the single-channel two-dimensional image, comprising: inputting the single-channel two-dimensional image into a key point detection model to obtain human key points, wherein the key point detection model is trained based on a third training sample, the third training sample comprises a second single-channel two-dimensional image and human key points corresponding to the second single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the second single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the second single-channel two-dimensional image; Based on the depth information corresponding to the human body key points in the single-channel two-dimensional image, converting the two-dimensional data of the human body key points in the single-channel two-dimensional image into three-dimensional point cloud data to obtain the three-dimensional point cloud data corresponding to the human body key points; The human body characteristic parameters are determined according to the three-dimensional point cloud data corresponding to the key points of the human body, wherein the human body characteristic parameters include: one or more of height, arm length, leg length or shoulder width.

6. A method for adjusting a seat, It is characterized in that include: Get the two-dimensional image captured by the TOF camera, where: The two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image; Detecting human key points in the single-channel two-dimensional image, comprising: inputting the single-channel two-dimensional image into a key point detection model to obtain human key points, wherein the key point detection model is trained based on a third training sample, the third training sample comprises a second single-channel two-dimensional image and human key points corresponding to the second single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the second single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the second single-channel two-dimensional image; Based on the depth information corresponding to the human body key points in the single-channel two-dimensional image, converting the two-dimensional data of the human body key points in the single-channel two-dimensional image into three-dimensional point cloud data to obtain the three-dimensional point cloud data corresponding to the human body key points; Determining human body characteristic parameters according to the three-dimensional point cloud data corresponding to the key points of the human body, wherein the human body characteristic parameters include: one or more of height, arm length, leg length or shoulder width; The parameters of the car seat are adjusted according to the human body characteristic parameters.

7. The method according to claim 6, It is characterized in that The method further comprises: Detecting a human body region in the single-channel two-dimensional image; Converting the two-dimensional data of the human body region in the single-channel two-dimensional image into three-dimensional point cloud data of the human body based on the depth information corresponding to the human body region in the single-channel two-dimensional image; Determine the body weight according to the three-dimensional point cloud data of the body; The car seat parameters are adjusted according to the body weight and the body characteristic parameters.

8. The method according to claim 6 or 7, It is characterized in that The TOF camera is located at the junction of the B-pillar and the roof of the car.

9. A weight detection device, It is characterized in that include: The acquisition unit is used to acquire a two-dimensional image collected by a time-of-flight TOF camera, wherein: The two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image; A processing unit for: Detecting a human body region in the single-channel two-dimensional image based on an image segmentation model, wherein the image segmentation model is obtained by training based on a first training sample, wherein the first training sample includes a first single-channel two-dimensional image and a human body region corresponding to the first single-channel two-dimensional image, and a grayscale value corresponding to a pixel in the first single-channel two-dimensional image is used to indicate depth information corresponding to the pixel in the first single-channel two-dimensional image; Converting the two-dimensional data of the human body region in the single-channel two-dimensional image into three-dimensional point cloud data of the human body based on the depth information corresponding to the human body region in the single-channel two-dimensional image; The three-dimensional point cloud data of the human body is input into a weight assessment model to obtain the human body weight.

10. The device according to claim 9, It is characterized in that The weight assessment model is obtained by training based on a second training sample, where the second training sample includes first human body three-dimensional point cloud data and a weight corresponding to the first human body three-dimensional point cloud data.

11. The device according to claim 10, It is characterized in that The single-channel two-dimensional image includes multiple frames of single-channel two-dimensional images, and The processing unit is specifically used for: Based on the depth information corresponding to the human body region in the multiple frames of single-channel two-dimensional images, converting the two-dimensional data of the human body region in the multiple frames of single-channel two-dimensional images into human body three-dimensional point cloud data to obtain multiple frames of human body three-dimensional point cloud data; The multiple frames of human body three-dimensional point cloud data are fused to obtain fused human body three-dimensional point cloud data.

12. The device according to any one of claims 9 to 11, It is characterized in that The processing unit is specifically used for: Uniformly refining the three-dimensional point cloud data of the human body to obtain sparse three-dimensional point cloud data of the human body; The body weight is determined according to the sparse three-dimensional point cloud data of the body.

13. A human body characteristic parameter detection device, It is characterized in that include: The acquisition unit is used to acquire a two-dimensional image collected by a time-of-flight TOF camera, wherein: The two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image; A processing unit for: Detecting human key points in the single-channel two-dimensional image, comprising: inputting the single-channel two-dimensional image into a key point detection model to obtain human key points, wherein the key point detection model is trained based on a third training sample, the third training sample comprises a second single-channel two-dimensional image and human key points corresponding to the second single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the second single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the second single-channel two-dimensional image; Based on the depth information corresponding to the human body key points in the single-channel two-dimensional image, converting the two-dimensional data of the human body key points in the single-channel two-dimensional image into three-dimensional point cloud data to obtain the three-dimensional point cloud data corresponding to the human body key points; The human body characteristic parameters are determined according to the three-dimensional point cloud data corresponding to the key points of the human body, wherein the human body characteristic parameters include: one or more of height, arm length, leg length or shoulder width.

14. A seat adjustment device, It is characterized in that include: The acquisition unit is used to acquire a two-dimensional image captured by a TOF camera, wherein: The two-dimensional image is a single-channel two-dimensional image, wherein the grayscale value corresponding to the pixel in the single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the single-channel two-dimensional image; A processing unit for: Detecting human key points in the single-channel two-dimensional image, comprising: inputting the single-channel two-dimensional image into a key point detection model to obtain human key points, wherein the key point detection model is trained based on a third training sample, the third training sample comprises a second single-channel two-dimensional image and human key points corresponding to the second single-channel two-dimensional image, and the grayscale value corresponding to the pixel in the second single-channel two-dimensional image is used to indicate the depth information corresponding to the pixel in the second single-channel two-dimensional image; Based on the depth information corresponding to the human body key points in the single-channel two-dimensional image, converting the two-dimensional data of the human body key points in the single-channel two-dimensional image into three-dimensional point cloud data to obtain the three-dimensional point cloud data corresponding to the human body key points; Determining human body characteristic parameters according to the three-dimensional point cloud data corresponding to the key points of the human body, wherein the human body characteristic parameters include: one or more of height, arm length, leg length or shoulder width; The parameters of the car seat are adjusted according to the human body characteristic parameters.

15. The device according to claim 14, It is characterized in that The processing unit is also used for: Detecting a human body region in the single-channel two-dimensional image; Converting the two-dimensional data of the human body region in the single-channel two-dimensional image into three-dimensional point cloud data of the human body based on the depth information corresponding to the human body region in the single-channel two-dimensional image; Determine the body weight according to the three-dimensional point cloud data of the body; The car seat parameters are adjusted according to the body weight and the body characteristic parameters.

16. The device according to claim 14 or 15, It is characterized in that The TOF camera is located at the junction of the B-pillar and the roof of the car.

17. A weight detection device, It is characterized in that The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 4.

18. A human body characteristic parameter detection device, It is characterized in that The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to claim 5.

19. A seat adjustment device, It is characterized in that The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 6 to 8.

20. A computer-readable storage medium, It is characterized in that The computer-readable medium stores a program code for execution by a device, the program code including a program for executing the method according to any one of claims 1 to 4 or 5 or 6 to 8.

21. A chip, It is characterized in that The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method as claimed in any one of claims 1 to 4 or 5 or 6 to 8.

Images