A method, device, equipment and medium for detecting seat information of a person in a vehicle
By combining the comprehensive judgment method of face and human body and the weighted binary graph algorithm, the problems of low recognition efficiency and poor accuracy in complex environments are solved, and high-accuracy occupants recognition and positioning are achieved, ensuring that Passengers in the vehicle are safe.
Patent Information
- Application Number
- CN202411017408.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-29
AI Technical Summary
Traditional passenger monitoring methods are inefficient and have high misjudgment rates, making it impossible to accurately identify passenger attributes, especially in complex in-vehicle environments.
Using a method that combines the comprehensive judgment of human face and human body, through the multimodal information fusion of human face and human body images, the weighted binary graph and KM algorithm are used to match, and the coordinate positioning of key points of human body is determined to determine the specific position of the occupant and the children. property.
Significantly improve the recall and accuracy of occupant identification, overcome identification failures caused by insufficient light and occlusion, accurately determine the occupant location, ensure the safety of child occupants, and provide valuable information for passenger management.
Smart Images

Figure CN118552941B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and medium for detecting seating information of a person in a vehicle. Background Art
[0002] With the rapid development of intelligent transportation systems (ITS), passenger monitoring and management inside vehicles has become an important technical means to ensure passenger safety, optimize passenger experience and improve vehicle management efficiency.
[0003] Traditional passenger monitoring methods mainly rely on manual observation or simple sensor detection. These methods have problems such as low efficiency, high misjudgment rate, and inability to accurately identify passenger attributes. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a method, device, equipment and medium for detecting seating information of occupants in a vehicle, which can improve the efficiency of detecting seating information of occupants in a vehicle.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides a method for detecting seat information of a person in a vehicle, comprising the following steps:
[0007] Acquire an in-vehicle video image of a target vehicle, and perform target detection processing on the in-vehicle video image to obtain at least one face image and at least one body image;
[0008] According to the in-car area information to which each human body image belongs, the human body image is cropped to obtain an optimized target human body image; wherein different in-car areas correspond to different cropping sizes;
[0009] Performing human body key point recognition processing on the target human body image to obtain target joint point information and label information in the target human body image; wherein the label information indicates whether the target human body image is a child;
[0010] Based on the label information in each of the target human images, and at least one of the target joint point information in the target human image and the face center point in the at least one face image, the seating information of the person in the target vehicle is determined.
[0011] In a second aspect, an embodiment of the present application further provides a device for detecting seat information of a person in a vehicle, the device comprising:
[0012] An acquisition module, used to acquire an in-vehicle video image of a target vehicle, and perform target detection processing on the in-vehicle video image to obtain at least one face image and at least one body image;
[0013] A cropping module is used to crop the human body image according to the in-vehicle area information to which each human body image belongs, so as to obtain an optimized target human body image; wherein different in-vehicle areas correspond to different cropping sizes;
[0014] A recognition module, used for performing human key point recognition processing on the target human image to obtain target joint point information and label information in the target human image; wherein the label information indicates whether the target human image is a child;
[0015] A determination module is used to determine the seating information of the person in the target vehicle based on the label information in each of the target human images, and at least one of the target joint point information in the target human image and the face center point in the at least one face image.
[0016] In the third aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the method for detecting seating information of a person in a vehicle as described in any one of the first aspects.
[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting seating information of a person in a vehicle as described in any one of the first aspects is executed.
[0018] The embodiments of the present application have the following beneficial effects:
[0019] By combining the method of comprehensive judgment of face and body, significant beneficial effects are achieved in the identification and positioning of occupants in the vehicle. Specifically, compared with a single face or body recognition method, this method has greatly improved the recall rate and accuracy. First, by combining the comprehensive information of face and body, the present application can more accurately identify occupants in a complex in-vehicle environment. Face information provides the facial features of the occupants, while body information provides more dimensional information such as the body shape and posture of the occupants. This fusion of multimodal information enables the system to overcome the limitations that may exist in a single information source when identifying occupants, such as failure of face recognition due to insufficient light, or inaccurate body recognition due to partial occupant body occlusion. Therefore, the comprehensive judgment method significantly improves the recall rate and accuracy of the system. Secondly, the present application uses a weighted bipartite graph and KM algorithm to match occupants and faces and bodies, further improving the matching accuracy. This method constructs a weighted adjacency matrix by calculating the distance or similarity between the headrest position and the face image and the key points of the human body. Then, the KM algorithm is used to find the optimal matching result on this adjacency matrix, that is, the sum of the weights of all edges that are successfully matched is minimized. This method not only takes into account the spatial position relationship between the occupant and the seat, but also takes into account other features between the occupant and the seat, thereby greatly improving the accuracy of matching. In addition, the present application uses the coordinate positioning of the key points of the human body to accurately determine where the occupant is sitting, rather than a simple regional determination. This method can accurately determine the specific position of the occupant and avoid the misjudgment that may be caused by regional determination. Finally, the present application can also give the child attributes of the occupant without increasing the time consumption. This is of great significance for vehicle safety monitoring and passenger management. By identifying child occupants, the system can automatically adjust the safety settings of the vehicle (such as the switch status of the airbag) to ensure the safety of child occupants. At the same time, the identification of child attributes can also provide valuable information for passenger management, such as the age distribution of passengers, the number of child passengers, etc. The realization of this function not only improves the intelligence level of the system, but also reduces resource consumption.
[0020] In summary, this application achieves significant beneficial effects in the identification and positioning of occupants in vehicles by combining the method of comprehensive determination of face and body, using weighted bipartite graph and KM algorithm for matching, determining the position of occupants by the coordinates of key points of the human body, and giving the child attributes of occupants. These beneficial effects not only improve the accuracy and reliability of the system, but also enhance the user experience and the safety of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 It is a flowchart of steps S101-S104 provided in an embodiment of the present application;
[0023] Figure 2 It is a flowchart of steps S201-S202 provided in an embodiment of the present application;
[0024] Figure 3 It is a flowchart of steps S301-S302 provided in an embodiment of the present application;
[0025] Figure 4 It is a flowchart of steps S401-S403 provided in an embodiment of the present application;
[0026] Figure 5 It is a flowchart of steps S501-S502 provided in an embodiment of the present application;
[0027] Figure 6 It is a flowchart of steps S601-S602 provided in an embodiment of the present application;
[0028] Figure 7 It is a flowchart of steps S701-S703 provided in an embodiment of the present application;
[0029] Figure 8 It is a flowchart of steps S801-S802 provided in an embodiment of the present application;
[0030] Fig. 9 It is a flowchart of steps S901-S902 provided in an embodiment of the present application;
[0031] Fig.10 It is a flowchart of steps S1001-S1003 provided in an embodiment of the present application;
[0032] Fig.11 It is a flowchart of steps S1101-S1102 provided in an embodiment of the present application;
[0033] Fig.12 It is a flowchart of steps S1201-S1203 provided in an embodiment of the present application;
[0034] Fig.13It is a flowchart of steps S1301-S1302 provided in an embodiment of the present application;
[0035] Fig.14 It is a structural schematic diagram of a device for detecting seat information of a person in a vehicle provided in an embodiment of the present application;
[0036] Fig.15 It is a schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0037] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0039] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0040] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0041] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0043] The acquisition, use and analysis of all user information involved in this application have been authorized by the user, and the implementation of this invention is legal use.
[0044] See also Figure 1 , Figure 1 is a flow chart of steps S101-S104 of the method for detecting the seat information of a person in a vehicle provided in an embodiment of the present application, which will be combined with Figure 1 Steps S101-S104 are shown for explanation.
[0045] Step S101, obtaining an in-vehicle video image of a target vehicle, and performing target detection processing on the in-vehicle video image to obtain at least one face image and at least one body image;
[0046] Step S102, performing image cropping processing on each human body image according to the in-vehicle region information to which the human body image belongs, to obtain an optimized target human body image; wherein different in-vehicle regions correspond to different cropping sizes;
[0047] Step S103, performing human body key point recognition processing on the target human body image to obtain target joint point information and label information in the target human body image; wherein the label information indicates whether the target human body image is a child;
[0048] Step S104, determining the seating information of the person in the target vehicle based on the label information in each of the target human images, and at least one of the target joint point information in the target human image and the face center point in the at least one face image.
[0049] The above exemplary steps of the embodiments of the present application are described below respectively.
[0050] In step S101, an in-vehicle video image of a target vehicle is acquired, and target detection processing is performed on the in-vehicle video image to obtain at least one face image and at least one body image.
[0051] Here, cameras installed inside the vehicle can be used to capture in-vehicle video images in real time. These cameras can be distributed in key locations such as the front row, back row, and side of the vehicle to ensure full coverage of the interior space. Based on the raw video data obtained by the camera, necessary preprocessing can be performed, such as denoising and contrast enhancement, to improve the accuracy of subsequent target detection.
[0052] The pre-processed in-car video images are subjected to target detection. This step mainly uses advanced machine learning algorithms such as deep learning to identify and classify objects in the image.
[0053] In the process of target detection, we focus on two types of targets: faces and bodies. Through the trained face detection model and body detection model, we can quickly and accurately locate the positions of faces and bodies in the image and extract the corresponding image areas.
[0054] For face detection, methods based on Haar features or deep learning can be used, such as MTCNN (Multi-task Cascaded Convolutional Networks), etc. These methods can accurately detect faces in complex backgrounds and provide information about the location and size of the faces.
[0055] For human detection, you can use methods based on HOG (Histogram of Oriented Gradients) features + SVM (Support Vector Machine), or target detection algorithms based on deep learning, such as Faster R-CNN, YOLO, etc. These methods can detect the human body in the image and give the position and bounding box information of the human body.
[0056] It should be noted that in some special cases (such as when there is occlusion), only face images or body images may be detected.
[0057] In step S102, according to the in-vehicle region information to which each human body image belongs, the human body image is cropped to obtain an optimized target human body image; wherein different in-vehicle regions correspond to different cropping sizes.
[0058] Here, due to the complex layout of the interior space of the vehicle, the size and proportion of the human body in different areas may be different in the image. Therefore, according to the information of the interior area to which each human body image belongs, targeted cropping processing can be performed to obtain an optimized target human body image, thereby improving the accuracy of subsequent analysis.
[0059] In the target detection stage, in addition to acquiring face and body images, it is also necessary to record the area information of each body image in the car. This information can be determined by the camera position, the internal layout of the vehicle, and the position relationship in the image.
[0060] Different cropping sizes are set according to the characteristics of different areas in the car and the actual size of the human body in the image. For example, the front area is closer to the camera, so the human body may be larger in the image, so a larger cropping size needs to be set; while the back area is farther away, so the human body may be smaller in the image, so a smaller cropping size needs to be set.
[0061] The original human images are cropped according to the in-car area information and the corresponding cropping size of each human image. The cropping process needs to ensure that the key information of the human image (such as the head, limbs, etc.) is not cropped, and the background and other irrelevant information are removed as much as possible.
[0062] After cropping, optimized target human images are obtained. These images only contain the human body area of interest and remove redundant information, which helps to improve the accuracy and efficiency of subsequent steps such as human key point recognition and label recognition.
[0063] By setting different cropping sizes for different in-car areas, it is possible to ensure that the optimized human image contains key information and remove redundant information, thereby improving the accuracy of subsequent analysis. The optimized human image only contains the area of interest, so the amount of calculation and time consumption in the subsequent processing can be reduced, and the overall processing efficiency can be improved. This solution takes into account the characteristics of different in-car areas and the actual size of the human body in the image, so it has strong adaptability and can adapt to vehicles of various models and internal layouts.
[0064] In step S103, human body key point recognition processing is performed on the target human body image to obtain target joint point information and label information in the target human body image; wherein the label information indicates whether the target human body image is a child.
[0065] Here, by identifying the key joints of the human body, the posture and movement of the occupants can be further analyzed. At the same time, determining whether the human image represents a child is also crucial to providing personalized services and ensuring the safety of the ride. This application embodiment will introduce how to perform human key point recognition processing on the target human image and obtain the target joint point information and label information (whether it is a child).
[0066] Use the trained human key point detection models (such as OpenPose, DeepCut, etc.) to identify the key points of the target human image. These models are usually based on deep learning technology and can accurately identify multiple key points in human images, such as the head, shoulders, elbows, wrists, hips, knees, ankles, etc.
[0067] Apply the human key point detection model to the target human image to obtain the position information (such as coordinate values) of each key point. This information will be used for subsequent analysis and processing.
[0068] Due to the complexity of the vehicle's internal environment, there may be factors such as occlusion and insufficient light that may lead to inaccurate key point recognition. Therefore, after obtaining the key point information, some optimization algorithms (such as interpolation, filtering, etc.) can be used to fine-tune the key point position to improve its accuracy.
[0069] The label information is used to characterize whether the target human image is a child. This can be achieved through a variety of methods, including but not limited to the following two: Method based on height ratio: By analyzing the ratio of the head to the whole body in the human image, it can be roughly determined whether it is a child. Generally speaking, the head of a child is larger than the whole body, so it can be determined by setting a certain threshold.
[0070] Machine learning-based methods: Train a specialized classifier (such as support vector machines, neural networks, etc.) to distinguish between human images of children and non-children. This method requires the collection of a large amount of training data with labels (children / non-children) and the training of the classifier to enable it to have recognition capabilities. In practical applications, the appropriate method can be selected to determine the label information according to the specific scenario and needs.
[0071] Finally, the identified target joint information and the determined label information are integrated and output. This information can be used as a basis for subsequent analysis and processing in areas such as ride safety monitoring and passenger services.
[0072] In the above-mentioned way, by adopting advanced human key point detection model and label information determination method, this solution can realize accurate recognition of human key points and reliable determination of label information. Considering the complexity of the vehicle's internal environment, through optimization algorithm and multiple determination methods, it can adapt to the needs of different scenarios. Due to the use of efficient image processing technology and machine learning algorithm, this solution can process and analyze the target human image in real time, meeting the real-time requirements in practical applications.
[0073] In step S104, based on the label information in each of the target human images, and at least one of the target joint point information in the target human image and the face center point in the at least one face image, the seating information of the person in the target vehicle is determined.
[0074] Here, it is necessary to comprehensively consider the label information of the target human image (such as whether it is a child), the target joint point information, and the position of the face center point in the face image to accurately determine the seating information of the person in the target vehicle.
[0075] For example, collect label information of each target human image (whether it is a child). Collect target joint point information in each target human image, such as the location of key points such as the head, shoulders, and hips. Collect the location information of the center point of the face in at least one face image.
[0076] Seat position inference Infer the seat position of the person in the vehicle based on the target joint point information in the target human image, especially the position of the hip key point. For example, if the hip key point is located in the front seat area, it is inferred that the person is sitting in the front seat. If the target human image contains multiple occupants, the relative positions of the joint points of each occupant are determined by comparing and analyzing the position relationship of the joint points of each occupant, thereby inferring the seating situation of the entire vehicle. Combined with label information Combine the label information (whether it is a child) with the seat position to form more detailed seat information of the person. For example, it can be marked which seat is occupied by a child to provide them with additional safety. Combined with face center information If available, the face center position information in the face image is used as an auxiliary means to verify or optimize the seat position inference. The face center usually corresponds to the head position of the occupant and can be used to verify whether the seat position inferred from the joint point information is accurate. In particular, when the joint point information is inaccurate due to occlusion or lighting problems, the face center information can serve as an important supplement.
[0077] Information integration and output All collected information is integrated to form a complete seating information report. The report can include the number of people in each seat, whether they are children, and possible postures and movements. The seating information report is output to the vehicle management system, driver display or other required places for subsequent analysis, processing or response.
[0078] In the above-mentioned way, by comprehensively utilizing the label information of the target human image, the target joint point information and the facial center point information in the facial image, this solution can accurately infer the seating information of the people in the vehicle. Not only the seating position of the occupants is taken into account, but also the identity of the occupants (such as children) and possible posture information are combined to provide more comprehensive data support for vehicle management. The embodiment of the present application adopts real-time image processing and analysis technology, which can update and output the seating information of the people in real time to meet the real-time requirements in practical applications.
[0079] In some embodiments, the step of acquiring an in-vehicle video image of a target vehicle includes:
[0080] An in-vehicle video image of a target vehicle is collected by an image acquisition device; the image acquisition device is installed at an interior rearview mirror or a reading light inside the target vehicle, and the orientation of the image acquisition device forms an acute angle with the horizontal direction of the target vehicle.
[0081] Here, in order to obtain the in-vehicle video image of the target vehicle, an image acquisition device needs to be installed. This solution will introduce in detail the installation position, orientation and angle of the image acquisition device with the horizontal direction of the target vehicle to ensure that a clear and comprehensive in-vehicle video image can be obtained.
[0082] The image acquisition device should be installed at the rearview mirror or reading light inside the target vehicle. Both locations are relatively central and have a wide field of view in the vehicle, which can cover most areas in the vehicle, thereby obtaining a comprehensive video image. When choosing to install it at the rearview mirror, it is necessary to ensure that the image acquisition device will not block the driver's line of sight, and the installation position is stable and reliable to prevent shaking or shifting during the vehicle's driving. When choosing to install it at the reading light, it is also necessary to ensure that the installation position is stable and reliable, and will not interfere with the normal use of other equipment in the vehicle.
[0083] The image acquisition device should be oriented toward the interior of the vehicle to ensure that the video image inside the vehicle can be captured. At the same time, in order to ensure the clarity and coverage of the video image, the image acquisition device should be oriented at an acute angle to the horizontal direction of the target vehicle. The specific size of this acute angle needs to be determined based on the internal structure of the vehicle and the performance of the image acquisition device. Generally speaking, a smaller angle can bring the image acquisition device closer to the center area of the vehicle, thereby obtaining a more comprehensive video image; but too small an angle may cause image distortion or distortion. Therefore, it is necessary to select a suitable angle while ensuring image clarity and coverage.
[0084] In the above method, by installing the image acquisition device at the rearview mirror or reading light and adjusting its direction and angle, a clear and comprehensive in-vehicle video image of the target vehicle can be obtained. This is of great significance for subsequent image processing, personnel seat information detection and other applications. In practical applications, it is necessary to select a suitable installation position and angle according to the internal structure of the vehicle and the performance of the image acquisition device, and perform necessary debugging to ensure its normal operation.
[0085] In some embodiments, the acute angle ranges from 20 degrees to 70 degrees.
[0086] Here, the acute angle is recommended to be in the range of 20 to 70 degrees. Within this angle range, the image acquisition device can capture sufficiently clear and wide in-vehicle images while avoiding image distortion or distortion caused by excessively large angles.
[0087] In some embodiments, see Figure 2 , Figure 2 It is a flowchart of steps S201-S202 provided in an embodiment of the present application. The target detection processing of the in-vehicle video image can be achieved through steps S201-S202, which will be explained in combination with each step.
[0088] In step S201, the in-vehicle video image is sampled to obtain a sampled image.
[0089] In step S202, the sampled image is input into a pre-trained target sampling model to obtain the at least one face image and the at least one body image, each of the at least one face image or the at least one body image corresponds to a position information, and the position information is represented as the horizontal and vertical coordinates of the upper left corner and the lower right corner of the image frame including the image.
[0090] In combination with steps S201-S202, it can be seen that in order to accurately identify faces and bodies from in-vehicle video images, the embodiment of the present application adopts a target detection processing scheme to sample and process in-vehicle video images and how to use a pre-trained target sampling model to extract face and body images and determine their location information.
[0091] First, the acquired in-car video images are sampled to obtain a series of sampled images. The purpose of sampling is to select representative images from continuous video frames to reduce the amount of calculation and processing time. The sampling strategy can be determined according to actual needs, such as sampling at a certain interval of frames, or sampling when key events (such as people entering or leaving) are detected. Input the sampled images into the pre-trained target sampling model. This model has been trained with a large amount of data and can recognize faces and human targets in images.
[0092] Target sampling models are usually based on deep learning techniques such as convolutional neural networks (CNN) or target detection algorithms (such as YOLO, SSD, etc.). These algorithms can accurately locate and identify target objects in complex backgrounds.
[0093] Extracting face and body images After the target sampling model processes the input sampled images, it outputs the detected target objects and their related information. In this scenario, we focus on face and body images. The model outputs at least one face image and at least one body image, and gives the location information corresponding to each image. The location information is usually expressed as the horizontal and vertical coordinates of the upper left corner and lower right corner of the image box (i.e., bounding box) that includes the image.
[0094] Position information is very important for subsequent analysis and processing. It can help us determine the specific position of the face and body in the car, so as to perform more accurate seat information detection or other related applications. In the embodiment of the present application, the position information is represented by the horizontal and vertical coordinates of the upper left corner and the lower right corner of the image frame. These coordinate values can be converted and adjusted according to the specific image resolution and coordinate system.
[0095] The above method, through sampling processing and pre-trained target sampling model, accurately extracts face and body images from the in-vehicle video images and gives their location information. This information provides important data support for subsequent applications such as seat information detection and behavior analysis. In practical applications, the sampling processing strategy and target sampling model can be selected and adjusted according to specific needs to obtain better results.
[0096] In some embodiments, see Figure 3 , Figure 3 It is a flow chart of steps S301-S302 provided in an embodiment of the present application. According to the in-vehicle area information to which each human body image belongs, image cropping is performed on the human body image to obtain an optimized target human body image. This can be achieved through steps S301-S302 and will be explained in combination with each step.
[0097] In step S301, the cropping size of each human body image is determined based on the in-vehicle area information to which each human body image belongs; wherein the in-vehicle area information includes a front row area and a rear row area, and the cropping size includes a first cropping size and a second cropping size, and the first cropping size of the human body images located in the front row area is the same, and the second cropping size of the human body images located in the rear row area is the same.
[0098] In step S302, each human body image is cropped based on the cropping size of each human body image.
[0099] Combining steps S301-S302, it can be seen that in order to further optimize the human images identified in the in-vehicle video images, it is necessary to crop the images according to the in-vehicle area information to which each human image belongs, so as to obtain a more accurate and useful target human image. This solution will introduce in detail how to determine the cropping size according to the in-vehicle area information, and crop each human image accordingly.
[0100] The in-vehicle area information usually includes the front area and the rear area. These areas can be divided according to the internal structure and seat layout of the vehicle. The front area usually includes the driver's seat and the co-pilot seat, and the rear area includes the rear passenger seat.
[0101] According to the in-car area information, we set different cropping sizes for human body images in different areas. These cropping sizes include the first cropping size and the second cropping size. The human body images in the front row area will use the same first cropping size, which is mainly determined by the layout of the front seats and the field of vision of the driver and passengers to ensure that important features such as the face and upper body are fully preserved. The human body images in the back row area will use the same second cropping size, which can be relatively small. Usually when collecting images, the images of people in the back row may be smaller due to "near big and far small". If the same cropping size is used as the front row, the human body images in the back row will have more useless information.
[0102] After that, the cropping area of each human image is determined based on the in-car area information and the corresponding cropping size to which each human image belongs. This is usually achieved by drawing a rectangular box corresponding to the cropping size on the original image. The cropping process is based on the determined cropping area, and each human image is cropped. This can be achieved through image processing software or algorithms, which removes the parts outside the cropping area from the image and only retains the parts within the cropping area. After the output optimized target human image is cropped, the optimized target human image is obtained. These images will only contain the parts of the human body within the specific area, and the size is more appropriate, which is convenient for subsequent analysis and processing.
[0103] The above method combines the in-car area information and a specific cropping size to crop the human body image identified in the in-car video image to obtain an optimized target human body image. This method can ensure that the image contains only useful information and has a more appropriate size, avoiding the problem of "large near and small far", and providing more accurate and effective data support for subsequent applications such as seat information detection and behavior analysis. In actual applications, the cropping size and area division can be adjusted according to specific needs to obtain better results.
[0104] In some embodiments, see Figure 4 , Figure 4 It is a flowchart of steps S401-S403 provided in an embodiment of the present application. The in-vehicle area information to which each human body image belongs can be determined through steps S401-S403, and will be explained in combination with each step.
[0105] In step S401, for each human body image, a first intersection area between each human body image and the front row area and a second intersection area between each human body image and the back row area are determined.
[0106] In step S402, a first ratio of the first intersection area to the front row area and a second ratio of the second intersection area to the back row area are determined.
[0107] In step S403, when the first ratio is greater than the second ratio, it is determined that the human body image is located in the front row, and when the second ratio is greater than the first ratio, it is determined that the human body image is located in the back row.
[0108] In combination with steps S401-S403, for each human body image, first determine its intersection area with the front row area and the back row area. The intersection area refers to the part of the human body image that overlaps with a specific area. Calculate a first ratio of the first intersection area to the front row area. This can be achieved by dividing the area or number of pixels of the first intersection area by the area or number of pixels of the front row area. At the same time, calculate a second ratio of the second intersection area to the back row area. This can also be achieved by dividing the area or number of pixels of the second intersection area by the area or number of pixels of the back row area.
[0109] If the first ratio is greater than the second ratio, it means that the intersection of the human body image and the front row area is larger, so it can be determined that the human body image is located in the front row. If the second ratio is greater than the first ratio, it means that the intersection of the human body image and the back row area is larger, so it can be determined that the human body image is located in the back row.
[0110] In the above method, the intersection area of the human body image and the front row area and the rear row area is calculated to determine the area information of the vehicle interior to which the human body image belongs. This method is simple and effective, and can accurately classify the human body image as the front row or the rear row, providing valuable information for subsequent processing and analysis.
[0111] In some embodiments, see Figure 5 , Figure 5 It is a flow chart of steps S501-S502 provided in an embodiment of the present application. Determining the cropping size of each human image based on the in-vehicle area information to which each human image belongs can be achieved through steps S501-S502, which will be explained in combination with each step.
[0112] In step S501, a preset area corresponding to each front seat and each rear seat is determined based on the seats in the vehicle.
[0113] In step S502, the cropping size is determined based on the preset area.
[0114] Here, in order to more accurately crop the human body image, a preset area is determined based on the seats in the car, and the cropping size is determined accordingly.
[0115] According to the seating layout in the vehicle, preset areas are defined for each front seat and each rear seat. These preset areas are usually set according to the actual position, size and field of view of the seat to ensure that the passenger's body parts can be completely covered.
[0116] Once the preset areas are determined, we can determine the cropping size of each human image based on these areas. The cropping size usually includes the width and height of the image, which should be able to completely cover the passenger's body part within the preset area while avoiding too much background information being included. For each human image, first determine the area in the car to which it belongs (front or back row), and then find the preset area for the corresponding area. Next, determine the cropping size based on the size of the preset area to ensure that the cropped image can fully display the passenger's body part.
[0117] When defining the preset areas, factors such as different vehicle models, different seat layouts, and passenger comfort need to be considered to ensure the accuracy and rationality of the preset areas.
[0118] In the above method, by defining a preset area based on the seats in the car and determining the cropping size of each human image accordingly, the human image can be cropped more accurately, providing valuable information for subsequent processing and analysis. This method is not only simple and easy to implement, but also has high accuracy and reliability.
[0119] In some embodiments, see Figure 6 , Figure 6 It is a flow chart of steps S601-S602 provided in an embodiment of the present application. The cropping of each human image based on the cropping size of each human image can be achieved through steps S601-S602, which will be explained in combination with each step.
[0120] In step S601, a third intersection area between each human body image and the preset area corresponding to the human body image is determined.
[0121] In step S602, a cropping center point is determined based on the intersection area, and a cropping area of a specified size is determined according to the cropping center point, and each human body image is cropped based on the cropping area.
[0122] Here, after determining the cropping size of each human image, each human image needs to be actually cropped. In order to ensure the accuracy and effectiveness of cropping, the cropping center point is determined based on the intersection area of the human image and the corresponding preset area, and the cropping area of the specified size is determined accordingly for cropping.
[0123] For each human image, we first determine the intersection area with the corresponding preset area. The intersection area refers to the overlapped part of the human image and the preset area, which represents the specific position of the human image in the preset area.
[0124] Next, a clipping center point is calculated based on the shape and size of the intersection area. This center point is usually located at the center of the intersection area, or is the most representative point calculated based on a specific algorithm (such as weighted average, etc.).
[0125] With the crop center point, we can determine a crop area of a specified size based on the previously determined crop size (i.e., the width and height of the crop area). This crop area is centered at the crop center point and extends to the specified width and height. Crop each human image Finally, we crop each human image based on the determined crop area. Cropping can be achieved through image processing software or algorithms, which will only retain the image portion within the crop area and remove the portion outside the crop area.
[0126] The above method determines the cropping center point based on the intersection area of the human body image and the corresponding preset area, and determines the cropping area of the specified size for cropping. This method can ensure that the cropped image can completely contain the human body part, and the size and position meet the preset requirements, providing high-quality data support for subsequent processing and analysis.
[0127] In some embodiments, the target joint point information in the target human body image is obtained by:
[0128] The target human image is input into a human key point detection model to obtain the target joint point information; wherein the in-vehicle area information includes a front row area and a rear row area, and the human key point detection model uses different loss functions for the front row area and the rear row area during training.
[0129] Here, the human key point detection model is a deep learning model that can automatically detect and locate the joints of key parts of the human body (such as head, hands, feet, etc.) from the input human image. These joint point information is crucial for subsequent posture analysis, behavior recognition and other tasks.
[0130] Due to the special in-car environment, the postures and perspectives of the front and back passengers may be different. Therefore, different loss functions are used for the front and back areas when training the human key point detection model.
[0131] In the car environment, due to the difference in camera viewing angle and passenger seat position, passengers close to the camera (such as the driver and co-pilot) occupy a larger pixel area in the image, while passengers far from the camera (such as rear passengers) occupy a smaller pixel area in the image. In particular, different loss functions are required for the problem of distinguishing children in the front and rear seats.
[0132] To solve this problem, the same loss function can be used for the front and back rows, but the loss function values should be calculated separately. For example, if both use MSE loss, there is a front row MSE loss function and a back row MSE loss function. By using different loss functions or weights for different areas or tasks, the training process of the model can be more accurately controlled to better adapt it to specific application scenarios and needs.
[0133] The above method can effectively solve the problem of "large objects near and small objects far away" in the in-car environment that affects the human key point detection model. Through data enhancement, multi-scale feature fusion, using different loss functions for different areas, and ensemble learning, we can improve the model's adaptability to passengers of different sizes, thereby more accurately locating the human joint point information.
[0134] In some embodiments, the label information in the target human image is obtained by:
[0135] The target human image is input into the human key point detection model to obtain the label information; wherein a judgment node is pre-added in the human key point detection model, and the judgment node judges whether the target human image is a child based on the detection result of the human key point detection model.
[0136] Here, when processing in-car videos or images, in addition to obtaining the key point information of the human body, it is sometimes necessary to determine whether the image contains a child. This is crucial for ensuring the safety of children, triggering specific alarms, or providing personalized services. To achieve this goal, a judgment node is pre-added to the human key point detection model, which can determine whether the target human image is a child based on the detection results of the model.
[0137] In order to determine whether the target human image is a child, we pre-added a judgment node to the human key point detection model. This judgment node makes a judgment based on the detection results of the model (i.e., joint point information). First, the human key point detection model will extract features from the input target human image, including detecting and locating the key joint points of the human body. Next, the judgment node will analyze the joint point information and extract features related to children. These features may include body proportions (such as head-to-body ratio, limb length ratio, etc.), relative positional relationships between joint points, movement patterns, etc. Based on the above feature analysis, the judgment node will apply a classifier (such as support vector machine, logistic regression, neural network, etc.) to determine whether the target human image is a child. This classifier will learn the differences between children and non-children during the training process and make accurate judgments based on this.
[0138] In the above method, by pre-adding judgment nodes in the human key point detection model, we can determine whether the target human image is a child while obtaining the human key point information. This method makes full use of the detection results of the human key point detection model and improves the accuracy and efficiency of child detection. This is of great significance in ensuring child safety and providing personalized services.
[0139] In some embodiments, before determining the seating information of the person in the target vehicle, the method further includes:
[0140] For the at least one face image and the at least one body image, respectively calculating the distance from each face image to each body image;
[0141] Based on the calculation result, the at least one face image and the at least one body image are matched to obtain the face-body matching result;
[0142] The method further comprises:
[0143] The seating information of the person in the target vehicle is determined based on the face and body matching result; wherein the face and body matching result includes at least one of a successful face and body match, only a face, and only a body.
[0144] Here, when determining the seating information of people in the target vehicle, in addition to considering the key point detection of the human body, face detection can also be combined to further improve the accuracy. Face detection can help us identify the specific people in the vehicle, while the key point detection of the human body can provide the posture and position information of the people. Therefore, this method first calculates the distance between the face image and the human body image, matches them, and then determines the seating information of the people based on the matching results.
[0145] Matching processing of face image and body image:
[0146] For at least one detected face image and at least one detected body image, we need to calculate the distance from each face image to each body image. This distance can be calculated based on the position, size or other relevant features of the image in space. Based on the distance calculated above, we match the face image and the body image. If the distance between a face image and a body image is less than a preset threshold, they are considered to be a pair of matching images. The results of the matching process will include three situations: successful face-body matching (i.e., matching face and body images are found), face only (i.e., face is detected but no matching body image is found), and body only (i.e., body is detected but no matching face image is found).
[0147] If the face and body images match successfully, we can determine the person's seat position based on the position and posture information of the body image. For example, if the body image is located in the front seat area, we can determine that the person is sitting in the front row.
[0148] If only a face image is detected but no matching human image is found, this may be because the person is blocked by other objects or the human detection algorithm fails to successfully identify the person. In this case, we can infer the possible location of the person based on the position and size of the face image, but it should be noted that the accuracy of this inference may be low. If only a human image is detected but no matching face image is found, this may be because the person is not facing the camera or the facial features are not obvious. In this case, we can still determine the person's seated position based on the human image, but cannot identify the specific person.
[0149] In the above method, by combining face detection and human key point detection and calculating the distance between them for matching processing, we can more accurately determine the seating information of the person in the target vehicle. This method not only improves the accuracy of detection, but also provides more information about the person in the car, which is helpful for subsequent safety monitoring and personalized services.
[0150] In some embodiments, see Figure 7 , Figure 7 It is a flow chart of steps S701-S703 provided in an embodiment of the present application, wherein for the at least one facial image and the at least one human body image, the distance from each facial image to each human body image is calculated respectively; based on the calculation result, the at least one facial image and the at least one human body image are matched to obtain the facial-human matching result, which can be achieved through steps S701-S703 and will be explained in combination with each step.
[0151] In step S701, a face set is constructed based on the at least one face image, and a body set is constructed based on the at least one body image.
[0152] In step S702, a first adjacency matrix is constructed based on the face set and the body set.
[0153] In step S703, a first matching process is performed on the at least one face image and the at least one body image based on the first adjacency matrix to obtain the face-body matching result.
[0154] Here, when determining the seating information of the person in the target vehicle, the face image and the body image are used for matching processing. This method combines the advantages of the two image types and improves the accuracy and reliability of matching. The embodiment of the present application first constructs a face set and a body set, then constructs a first adjacency matrix based on the two sets, and performs matching processing of the face and body images based on the matrix.
[0155] First, from the face images obtained from the target vehicle, we extract features and construct a face set. Similarly, we extract features from the body images and construct a body set. Each element in these sets represents an independent face or body image. Next, based on the face set and the body set, we construct a first adjacency matrix. This matrix is used to represent the potential relationship between face images and body images. Each element in the matrix represents the similarity or distance between a face image and a body image. This can be achieved by calculating the Euclidean distance, cosine similarity or other metrics between image features. Finally, using the constructed first adjacency matrix, we match the face image and the body image. This usually involves setting a threshold. When an element in the matrix (i.e., the similarity or distance between the face image and the body image) is less than the threshold, we consider the pair of images to be matched. By traversing all elements of the matrix, we can get all the face and body matching results.
[0156] In the above manner, by constructing a face set and a body set, and constructing a first adjacency matrix based on them for matching processing, a face image can be effectively matched with a body image. This method not only improves the accuracy and reliability of matching, but also provides more information about the people in the target vehicle, which is helpful for subsequent safety monitoring and personalized services.
[0157] In some embodiments, see Figure 8 , Figure 8It is a flow chart of steps S801-S802 provided in an embodiment of the present application. The first matching processing is performed on the at least one face image and the at least one body image based on the first adjacency matrix to obtain the face and body matching result. It can be achieved through steps S801-S802 and will be explained in combination with each step.
[0158] In step S801, a first top label is assigned to each element in the face set and a second top label is assigned to each element in the body set.
[0159] In step S802, a complete match is obtained between each element in the face set and each element in the body set, wherein the sum of weights of all successfully matched edges is minimal, and the weight of each edge is the distance between the first and second top labels of the two elements corresponding to the edge.
[0160] The first top label is assigned to each element in the face set (i.e., each face image). This top label can be some characteristic value of the face image, such as one or more components in the characteristic vector, or some measurement based on the position, size, and other information of the face image in the image. Similarly, the second top label is assigned to each element in the human body set (i.e., each human body image). This top label can be the key point position information, posture information of the human body image, or some measurement based on the position, size, and other information of the human body image in the image.
[0161] Based on the first and second top marks of the elements in the face set and the body set, a first adjacency matrix is constructed. Each row of the matrix represents a face image, and each column represents a body image. The elements in the matrix (i.e., the weights of the edges) represent the similarity or distance between the corresponding face images and body images. This distance can be obtained by calculating the Euclidean distance, cosine similarity, or other measurement methods between the two top marks.
[0162] Next, we need to find a complete match between the face set and the body set, that is, each face image corresponds to a body image, and each body image also corresponds to a face image. This match needs to meet one condition: the sum of the weights of all edges that are successfully matched is the smallest. This is actually a typical weighted bipartite graph minimum matching problem, which can be solved using the Hungarian algorithm (Kuhn-Munkres algorithm) or the KM algorithm. In the solution process, we traverse all elements of the first adjacency matrix (that is, all possible edges) and try to find a complete match that meets the above conditions. For each edge, we check its weight (that is, the distance between the corresponding face image and body image), and select the edge with the smallest weight to add to the match. In the end, the matching result we get is the face-body matching result, which includes at least one of the following situations: successful face-body matching, only face, and only body.
[0163] Based on the face and body matching results, we can further determine the seating information of the person in the target vehicle. For example, if a face image successfully matches a body image, and the body image is located in the front seat area, we can determine that the person is sitting in the front row. If there is only a face image but no matching body image, we can infer the possible location of the person based on the position and size of the face image. This information is of great significance for applications such as vehicle safety monitoring and passenger recognition.
[0164] In some embodiments, see Fig. 9 , Fig. 9 It is a flow chart of steps S901-S902 provided in an embodiment of the present application, wherein the seating information of the person in the target vehicle is determined based on the label information in each of the target human images, the target joint point information in the target human image and at least one of the face center points in the at least one face image, which can be achieved through steps S901-S902 and will be explained in conjunction with each step.
[0165] In step S901, the human body sitting position is determined based on the target joint point information in the target human body image, and the face sitting position is determined based on the face center point in the at least one face image.
[0166] In step S902, based on the tag information, at least one of the human body seating position and the face seating position, the seating information of the person in the target vehicle is determined.
[0167] Here, firstly, the target human body images are obtained from the camera or sensor in the target vehicle. These images will be used as input data for subsequent processing. The target human body images are processed using the human body key point detection model to extract the key joint point information of the target human body. These joint point information includes the position and posture information of key parts such as the head, shoulders, elbows, and knees. At the same time, at least one face image is processed using the face detection algorithm to determine the position of the face center point in each face image. The face center point is usually located in the central area of the face image and can accurately reflect the facial orientation and position of the passenger.
[0168] Based on the extracted target joint point information, the seating position of the human body can be inferred. For example, if the detected human joint point information is concentrated in the front seat area, it can be determined that the person is sitting in the front row. Similarly, if the joint point information is concentrated in the back seat area, it can be determined that the person is sitting in the back row. The position information of the center point of the face can also be used to assist in determining the seating position of the passenger. For example, if the detected center point of the face is located in the front row and its position corresponds to the front seat area, it can be determined that the passenger is sitting in the front row. Similarly, if the center point of the face is located in the back row, it can be determined that the passenger is sitting in the back row.
[0169] In addition to directly determining the seat location based on human body and face information, the accuracy can be further improved by combining the label information in the target human image. The label information may include the passenger's age, gender, identity and other characteristics, which can be used to assist in determining the passenger's seat location.
[0170] Specifically, the possible seating position of a passenger can be determined based on the passenger's identity or age characteristics in the tag information. For example, if the tag information shows that a passenger is a child, then it can be given priority to match it with a child seat or a rear seat area. Similarly, if the tag information shows that a passenger is a driver or a front passenger, then it can be matched with a front seat area.
[0171] Finally, the seating information of the person in the target vehicle is determined by combining at least one of the following factors: the seating position of the person, the seating position of the face, and the tag information. This information will be used in subsequent vehicle management, safety monitoring, and personalized services.
[0172] In some embodiments, see Fig.10 , Fig.10 It is a flowchart of steps S1001-S1003 provided in an embodiment of the present application. The human body sitting position can be obtained through steps S1001-S1003, which will be explained in combination with each step.
[0173] In step S1001, for each seat in the vehicle, the seat position information of each seat is determined, and a seat set is constructed based on the seat position information of each seat; wherein each element in the seat set represents the seat position information of a seat.
[0174] In step S1002, a joint point set is constructed based on the joint point information; wherein each element in the joint point set represents a joint point corresponding to a target human body image.
[0175] In step S1003, a second matching process is performed based on the seat set and the joint point set to obtain a first matching result, and the first matching result is used as the human body seating position.
[0176] Here, first, for each seat in the car (such as the front driver's seat, the front passenger seat, the back seat, etc.), its seat position information is determined. This information may include the position of the seat in the vehicle coordinate system (such as X, Y, Z coordinates), the direction and size of the seat, etc. Based on the seat position information of each seat, a seat set is constructed. Each element in the set represents the seat position information of a seat and may contain a unique identifier of the seat (such as seat number, type, etc.).
[0177] The target human image is processed by the human key point detection algorithm to extract the joint point information of each person. The joint point information includes the position and posture information of key parts such as the head, shoulders, elbows, and knees. The extracted joint point information is grouped according to the human image, and a joint point set is constructed. Each element in the set represents the joint point information corresponding to a target human image, which may include the unique identifier of the human body (such as the detection number) and the specific position information of the joint point.
[0178] For each seat position information in the seat set and each joint point information in the joint point set, the matching degree between them is calculated. The matching degree can be calculated based on a variety of factors, such as the relative distance between the joint point position and the seat position, the consistency between the joint point orientation and the seat orientation, the matching degree between the joint point distribution and the seat size, etc. According to the calculated matching degree, the second matching process is performed. This can be achieved by using optimization methods such as greedy algorithm, dynamic programming, Hungarian algorithm, etc. The purpose is to find a matching method so that all relevant node information can match the seat position information and the total matching degree is the highest. After the second matching process, the first matching result is obtained. The result contains the matching relationship between each joint point information and its corresponding seat position information. According to the first matching result, the seat position of each human body in the vehicle can be determined. Specifically, the seat position of the person can be determined based on the seat position information that successfully matches a certain joint point information. If a certain joint point information does not successfully match any seat position, the seat position of the person will be marked as unknown or requires further confirmation.
[0179] The above method can accurately determine the seat position of the person in the target vehicle based on the layout of the seats inside the vehicle and the joint point information of the passengers. This is of great significance for applications such as vehicle safety monitoring and passenger identification.
[0180] In some embodiments, see Fig.11 , Fig.11It is a flow chart of steps S1101-S1102 provided in an embodiment of the present application. The second matching process based on the seat set and the joint point set can be implemented through steps S1101-S1102, which will be explained in combination with each step.
[0181] In step S1101, a second adjacency matrix is constructed based on the seat set and the joint point set.
[0182] In step S1102, a complete match is obtained between each element in the seat set and each element in the joint point set based on the second adjacency matrix, wherein the sum of the weights of all successfully matched edges is minimal, and the weight of each edge is the distance between the two elements corresponding to the edge.
[0183] In order to more accurately determine the seat position of the person in the target vehicle, we will use the seat set and the joint point set to construct a second adjacency matrix and use this matrix to find the optimal matching relationship. First, we have a seat set and a joint point set. Each element in the seat set contains the position information of the seat, while each element in the joint point set contains the joint point information of a human image. For each seat position in the seat set (which we call "seat element") and each joint point in the joint point set (which we call "joint point element"), we need to calculate the distance between them. This distance can be their actual physical distance in the vehicle coordinate system, or it can be the similarity or difference calculated based on some feature. Fill all the calculated distance values into a matrix, which is the second adjacency matrix. The rows of the matrix represent the elements in the seat set, and the columns represent the elements in the joint point set. Each element in the matrix (i.e., the weight of the edge) represents the distance between the seat element and the joint point element in the corresponding row and column.
[0184] Our goal is to find a matching method so that each element in the seat set matches an element in the joint point set, and the sum of the weights of all successfully matched edges is minimized. This is actually a typical weighted bipartite graph minimum matching problem. In order to find such an optimal matching, we can use classic algorithms such as the Hungarian algorithm (Kuhn-Munkres algorithm) or the KM algorithm. These algorithms can find a complete matching in a given weighted bipartite graph so that the sum of the weights of all successfully matched edges is minimized. After running the algorithm, a matching result will be obtained, that is, each seat element is matched with a joint point element, and these matches meet the condition of the minimum sum of weights.
[0185] Finally, based on the matching results, we can determine the seat position of each person in the vehicle. Specifically, if a seat element matches a joint element, we can assume that the passenger in the human image corresponding to the joint element is sitting on that seat. If a seat element does not match any joint element, or a joint element does not match any seat element, we can assume that these positions are empty or need further confirmation.
[0186] In some embodiments, see Fig.12 , Fig.12 It is a flowchart of steps S1201-S1203 provided in an embodiment of the present application. The face seating position can be obtained through steps S1201-S1203, which will be explained in combination with each step.
[0187] In step S1201, for each seat in the vehicle, the headrest position information of each seat is determined, and a headrest set is constructed based on the headrest position information of each seat; wherein each element in the headrest set represents the headrest position information of a seat.
[0188] In step S1202, a face set is constructed based on each face image; wherein each element in the face set represents a face image.
[0189] In step S1203, a third matching process is performed based on the headrest set and the face set to obtain a second matching result, and the second matching result is used as the seat position of the face.
[0190] Here, first, for each seat in the car (such as the front driver's seat, the front passenger seat, the back seat, etc.), determine the position information of the headrest. This information may include the position of the headrest in the vehicle coordinate system (such as X, Y, Z coordinates), orientation, etc. The headrest is usually an important component of the vehicle seat, and its position is closely related to the position of the human head. Based on the headrest position information of each seat, a headrest set is constructed. Each element in the set represents the headrest position information of a seat and may contain a unique identifier of the seat (such as seat number, type, etc.).
[0191] The facial images are acquired through the camera or other image acquisition devices in the car. These images will be used for subsequent face matching processing. The acquired facial images are grouped and identified to build a face set. Each element in the set represents an independent facial image and may contain the unique identifier of the face (such as the detection number) and related image data.
[0192] For each headrest position information in the headrest set and each face image in the face set, the matching degree between them is calculated. The matching degree can be calculated based on a variety of factors, such as the distance between the center point of the face in the face image and the headrest position, the consistency of the face orientation and the headrest orientation, the matching degree between the face size and the headrest size, etc. The specific calculation method can be selected and adjusted according to the actual situation. Based on the calculated matching degree, a third matching process is performed. This can be achieved by using optimization methods such as greedy algorithm and Hungarian algorithm, the purpose is to find a matching method so that each face image can match the headrest position information, and the total matching degree is the highest or meets a certain threshold condition. After the third matching process, a second matching result is obtained. The result contains the matching relationship between each face image and its corresponding headrest position information. According to the second matching result, the seat position of each face in the vehicle can be determined. Specifically, the seat corresponding to a face image can be determined based on the headrest position information that successfully matches the face image. If a face image does not successfully match any headrest position information, further analysis or confirmation of the seat position of the passenger may be required.
[0193] The above method can more accurately determine the seating position of people in the target vehicle based on facial information and headrest position information, providing strong support for vehicle safety monitoring and passenger management.
[0194] In some embodiments, see Fig.13 , Fig.13 It is a flowchart of steps S1301-S1302 provided in an embodiment of the present application. The third matching process based on the headrest set and the face set can be implemented through steps S1301-S1302, which will be explained in combination with each step.
[0195] In step S1301, a third adjacency matrix is constructed based on the headrest set and the face set.
[0196] In step S1302, a complete match is obtained between each element in the headrest set and each element in the face set based on the third adjacency matrix, wherein the sum of weights of all successfully matched edges is minimal, and the weight of each edge is the distance between the two elements corresponding to the edge.
[0197] Here, each element in the headrest set contains the headrest position information of each seat in the vehicle, while each element in the face set contains the information of a face image. For each headrest position in the headrest set (which we call a "headrest element") and each face image in the face set (which we call a "face element"), we need to calculate the distance between them. This distance can be their actual physical distance in the vehicle coordinate system, or it can be the similarity or difference calculated based on a certain feature. Fill all the calculated distance values into a matrix, which is the third adjacency matrix. The rows of the matrix represent the elements in the headrest set, and the columns represent the elements in the face set. Each element in the matrix (i.e., the weight of the edge) represents the distance between the headrest element and the face element in the corresponding row and column.
[0198] Similarly, our goal is to find a matching method so that each element in the headrest set matches an element in the face set, and the sum of the weights of all edges that are successfully matched is minimized. This is actually a typical weighted bipartite graph minimum matching problem. Classic algorithms such as the Hungarian algorithm (Kuhn-Munkres algorithm) can also be used. These algorithms can find a complete matching in a given weighted bipartite graph so that the sum of the weights of all edges that are successfully matched is minimized. After running the algorithm, we will get a matching result, that is, each headrest element matches a face element, and these matches meet the condition of the minimum sum of weights.
[0199] Finally, based on the matching results, we can determine the seating position of each face in the vehicle. Specifically, if a headrest element matches a facial element successfully, then we can assume that the passenger in the facial image corresponding to the facial element is sitting in the seat corresponding to that headrest. If a headrest element does not match any facial element, or a facial element does not match any headrest element, then we can assume that these positions are empty or require further confirmation. In this way, we can more accurately determine the seating position of passengers in the target vehicle based on the headrest position and facial information, providing strong support for vehicle safety monitoring and passenger management.
[0200] In summary, the embodiments of the present application have the following beneficial effects:
[0201] By combining the method of comprehensive judgment of face and body, significant beneficial effects are achieved in the identification and positioning of occupants in the vehicle. Specifically, compared with a single face or body recognition method, this method has greatly improved the recall rate and accuracy. First, by combining the comprehensive information of face and body, the present application can more accurately identify occupants in a complex in-vehicle environment. Face information provides the facial features of the occupants, while body information provides more dimensional information such as the body shape and posture of the occupants. This fusion of multimodal information enables the system to overcome the limitations that may exist in a single information source when identifying occupants, such as failure of face recognition due to insufficient light, or inaccurate body recognition due to partial occupant body occlusion. Therefore, the comprehensive judgment method significantly improves the recall rate and accuracy of the system. Secondly, the present application uses a weighted bipartite graph and KM algorithm to match occupants and faces and bodies, further improving the matching accuracy. This method constructs a weighted adjacency matrix by calculating the distance or similarity between the headrest position and the face image and the key points of the human body. Then, the KM algorithm is used to find the optimal matching result on this adjacency matrix, that is, the sum of the weights of all edges that are successfully matched is minimized. This method not only takes into account the spatial position relationship between the occupant and the seat, but also takes into account other features between the occupant and the seat, thereby greatly improving the accuracy of matching. In addition, the present application uses the coordinate positioning of the key points of the human body to accurately determine where the occupant is sitting, rather than a simple regional determination. This method can accurately determine the specific position of the occupant and avoid the misjudgment that may be caused by regional determination. Finally, the present application can also give the child attributes of the occupant without increasing the time consumption. This is of great significance for vehicle safety monitoring and passenger management. By identifying child occupants, the system can automatically adjust the safety settings of the vehicle (such as the switch status of the airbag) to ensure the safety of child occupants. At the same time, the identification of child attributes can also provide valuable information for passenger management, such as the age distribution of passengers, the number of child passengers, etc. The realization of this function not only improves the intelligence level of the system, but also reduces resource consumption.
[0202] In summary, this application achieves significant beneficial effects in the identification and positioning of occupants in vehicles by combining the method of comprehensive determination of face and body, using weighted bipartite graph and KM algorithm for matching, determining the position of occupants by the coordinates of key points of the human body, and giving the child attributes of occupants. These beneficial effects not only improve the accuracy and reliability of the system, but also enhance the user experience and the safety of the vehicle.
[0203] Based on the same inventive concept, an embodiment of the present application also provides a device for detecting the seating information of occupants in a vehicle corresponding to the method for detecting the seating information of occupants in a vehicle in the first embodiment. Since the principle of solving the problem by the device in the embodiment of the present application is similar to that of the above-mentioned method for detecting the seating information of occupants in a vehicle, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0204] like Fig.14 As shown, Fig.14 1 is a schematic diagram of the structure of the vehicle seat information detection device 1400 provided in an embodiment of the present application. The vehicle seat information detection device 1400 includes:
[0205] The acquisition module 1401 is used to acquire an in-vehicle video image of a target vehicle, and perform target detection processing on the in-vehicle video image to obtain at least one face image and at least one body image;
[0206] The cropping module 1402 is used to perform image cropping processing on each human body image according to the in-vehicle region information to which the human body image belongs, so as to obtain an optimized target human body image; wherein different in-vehicle regions correspond to different cropping sizes;
[0207] The recognition module 1403 is used to perform human key point recognition processing on the target human image to obtain target joint point information and label information in the target human image; wherein the label information indicates whether the target human image is a child;
[0208] The determination module 1404 is used to determine the seating information of the person in the target vehicle based on the label information in each of the target human images, and at least one of the target joint point information in the target human image and the face center point in the at least one face image.
[0209] Those skilled in the art should understand that Fig.14 The functions implemented by each unit in the vehicle occupant seating information detection device 1400 shown can be understood by referring to the relevant description of the aforementioned vehicle occupant seating information detection method. Fig.14 The functions of the various units in the vehicle occupant seating information detection device 1400 shown can be implemented by a program running on a processor, or by a specific logic circuit.
[0210] In a possible implementation, the acquisition module 1401 acquires the in-vehicle video image of the target vehicle, including:
[0211] An in-vehicle video image of a target vehicle is collected by an image acquisition device; the image acquisition device is installed at an interior rearview mirror or a reading light inside the target vehicle, and the orientation of the image acquisition device forms an acute angle with the horizontal direction of the target vehicle.
[0212] In a possible implementation, the acute angle ranges from 20 degrees to 70 degrees.
[0213] In a possible implementation, the acquisition module 1401 performs target detection processing on the in-vehicle video image, including:
[0214] Sampling the in-vehicle video image to obtain a sampled image;
[0215] The sampling image is input into a pre-trained target sampling model to obtain the at least one face image and the at least one body image, each of the at least one face image or the at least one body image corresponds to a position information, and the position information is represented as the horizontal and vertical coordinates of the upper left corner and the lower right corner of the image frame including the image.
[0216] In a possible implementation, the cropping module 1402 performs image cropping processing on each human body image according to the in-vehicle region information to which the human body image belongs, to obtain an optimized target human body image, including:
[0217] Determining the cropping size of each human body image based on the in-vehicle area information to which each human body image belongs; wherein the in-vehicle area information includes a front row area and a rear row area, the cropping size includes a first cropping size and a second cropping size, the first cropping size of the human body images located in the front row area is the same, and the second cropping size of the human body images located in the rear row area is the same;
[0218] Each human body image is cropped based on the cropping size of each human body image.
[0219] In a possible implementation manner, the cropping module 1402 determines the in-vehicle region information to which each human body image belongs in the following manner:
[0220] For each of the human body images, determining a first intersection area between the human body image and the front row area and a second intersection area between the human body image and the back row area;
[0221] Determine a first ratio of the first intersection area to the front area and a second ratio of the second intersection area to the back area;
[0222] When the first ratio is greater than the second ratio, it is determined that the human body image is located in the front row, and when the second ratio is greater than the first ratio, it is determined that the human body image is located in the back row.
[0223] In a possible implementation manner, the cropping module 1402 determines the cropping size of each human image based on the in-vehicle region information to which each human image belongs, including:
[0224] Determine a preset area corresponding to each front seat and each rear seat based on the seats in the vehicle;
[0225] The cropping size is determined based on the preset area.
[0226] In a possible implementation manner, the cropping module 1402 performs cropping processing on each human image based on the cropping size of each human image, including:
[0227] Determine a third intersection area of each human body image and the preset area corresponding to the human body image;
[0228] A cropping center point is determined based on the intersection area, and a cropping area of a specified size is determined according to the cropping center point, and each human body image is cropped based on the cropping area.
[0229] In a possible implementation manner, the target joint point information in the target human body image is obtained in the following manner:
[0230] The target human image is input into a human key point detection model to obtain the target joint point information; wherein the in-vehicle area information includes a front row area and a rear row area, and the human key point detection model uses different loss functions for the front row area and the rear row area during training.
[0231] In a possible implementation manner, the label information in the target human image is obtained in the following manner:
[0232] The target human image is input into the human key point detection model to obtain the label information; wherein a judgment node is pre-added in the human key point detection model, and the judgment node judges whether the target human image is a child based on the detection result of the human key point detection model.
[0233] In a possible implementation manner, before the determination module 1404 determines the seating information of the person in the target vehicle, the method further includes:
[0234] For the at least one face image and the at least one body image, respectively calculating the distance from each face image to each body image;
[0235] Based on the calculation result, the at least one face image and the at least one body image are matched to obtain the face-body matching result;
[0236] The determination module 1404 further includes:
[0237] The seating information of the person in the target vehicle is determined based on the face and body matching result; wherein the face and body matching result includes at least one of a successful face and body match, only a face, and only a body.
[0238] In a possible implementation, the determining module 1404 calculates the distance between each face image and each body image respectively for the at least one face image and the at least one body image; and performs matching processing on the at least one face image and the at least one body image based on the calculation result to obtain the face-body matching result, including:
[0239] constructing a face set based on the at least one face image, and constructing a body set based on the at least one body image;
[0240] Constructing a first adjacency matrix based on the face set and the body set;
[0241] A first matching process is performed on the at least one face image and the at least one body image based on the first adjacency matrix to obtain the face-body matching result.
[0242] In a possible implementation, the determining module 1404 performs a first matching process on the at least one face image and the at least one body image based on the first adjacency matrix to obtain the face-body matching result, including:
[0243] Assigning a first top label to each element in the face set and assigning a second top label to each element in the body set;
[0244] A complete match is obtained between each element in the face set and each element in the body set, wherein the sum of weights of all successfully matched edges is the smallest, and the weight of each edge is the distance between the first and second top marks of the two elements corresponding to the edge.
[0245] In a possible implementation, the determination module 1404 determines the seating information of the person in the target vehicle based on the label information in each of the target human images, and at least one of the target joint point information in the target human image and the face center point in the at least one face image, including:
[0246] Determine a human body seating position based on target joint point information in the target human body image, and determine a human face seating position based on a human face center point in the at least one human face image;
[0247] Based on the tag information, at least one of the human body seating position and the face seating position, the seating information of the person in the target vehicle is determined.
[0248] In a possible implementation manner, the human body seat position is obtained by:
[0249] For each seat in the vehicle, seat position information of each seat is determined, and a seat set is constructed based on the seat position information of each seat; wherein each element in the seat set represents the seat position information of one seat;
[0250] Constructing a joint point set based on the joint point information; wherein each element in the joint point set represents a joint point corresponding to a target human body image;
[0251] A second matching process is performed based on the seat set and the joint point set to obtain a first matching result, and the first matching result is used as the human body seating position.
[0252] In a possible implementation, the determining module 1404 performs a second matching process based on the seat set and the joint point set, including:
[0253] Constructing a second adjacency matrix based on the seat set and the joint point set;
[0254] Based on the second adjacency matrix, a complete match is obtained between each element in the seat set and each element in the joint point set, wherein the sum of weights of all successfully matched edges is the smallest, and the weight of each edge is the distance between the two elements corresponding to the edge.
[0255] In a possible implementation, the face seating position is obtained in the following manner:
[0256] For each seat in the vehicle, determine the headrest position information of each seat, and construct a headrest set based on the headrest position information of each seat; wherein each element in the headrest set represents the headrest position information of one seat;
[0257] Constructing a face set based on each face image; wherein each element in the face set represents a face image;
[0258] A third matching process is performed based on the headrest set and the face set to obtain a second matching result, and the second matching result is used as the seat position of the face.
[0259] In a possible implementation, the determination module 1404 performs a third matching process based on the headrest set and the face set, including:
[0260] Constructing a third adjacency matrix based on the headrest set and the face set;
[0261] Based on the third adjacency matrix, a complete match is obtained between each element in the headrest set and each element in the face set, wherein the sum of weights of all successfully matched edges is the smallest, and the weight of each edge is the distance between the two elements corresponding to the edge.
[0262] The above-mentioned vehicle seat information detection device has the following beneficial effects:
[0263] By combining the method of comprehensive judgment of face and body, significant beneficial effects are achieved in the identification and positioning of occupants in the vehicle. Specifically, compared with a single face or body recognition method, this method has greatly improved the recall rate and accuracy. First, by combining the comprehensive information of face and body, the present application can more accurately identify occupants in a complex in-vehicle environment. Face information provides the facial features of the occupants, while body information provides more dimensional information such as the body shape and posture of the occupants. This fusion of multimodal information enables the system to overcome the limitations that may exist in a single information source when identifying occupants, such as failure of face recognition due to insufficient light, or inaccurate body recognition due to partial occupant body occlusion. Therefore, the comprehensive judgment method significantly improves the recall rate and accuracy of the system. Secondly, the present application uses a weighted bipartite graph and KM algorithm to match occupants and faces and bodies, further improving the matching accuracy. This method constructs a weighted adjacency matrix by calculating the distance or similarity between the headrest position and the face image and the key points of the human body. Then, the KM algorithm is used to find the optimal matching result on this adjacency matrix, that is, the sum of the weights of all edges that are successfully matched is minimized. This method not only takes into account the spatial position relationship between the occupant and the seat, but also takes into account other features between the occupant and the seat, thereby greatly improving the accuracy of matching. In addition, the present application uses the coordinate positioning of the key points of the human body to accurately determine where the occupant is sitting, rather than a simple regional determination. This method can accurately determine the specific position of the occupant and avoid the misjudgment that may be caused by regional determination. Finally, the present application can also give the child attributes of the occupant without increasing the time consumption. This is of great significance for vehicle safety monitoring and passenger management. By identifying child occupants, the system can automatically adjust the safety settings of the vehicle (such as the switch status of the airbag) to ensure the safety of child occupants. At the same time, the identification of child attributes can also provide valuable information for passenger management, such as the age distribution of passengers, the number of child passengers, etc. The realization of this function not only improves the intelligence level of the system, but also reduces resource consumption.
[0264] In summary, this application achieves significant beneficial effects in the identification and positioning of occupants in vehicles by combining the method of comprehensive determination of face and body, using weighted bipartite graph and KM algorithm for matching, determining the position of occupants by the coordinates of key points of the human body, and giving the child attributes of occupants. These beneficial effects not only improve the accuracy and reliability of the system, but also enhance the user experience and the safety of the vehicle.
[0265] like Fig.15 As shown, Fig.15 The present invention provides a schematic diagram of the structure of an electronic device 1500, wherein the electronic device 1500 includes:
[0266] A processor 1501, a storage medium 1502 and a bus 1503, wherein the storage medium 1502 stores machine-readable instructions executable by the processor 1501. When the electronic device 1500 is running, the processor 1501 communicates with the storage medium 1502 via the bus 1503, and the processor 1501 executes the machine-readable instructions to perform the steps of the method for detecting seating information of a person in a vehicle described in the embodiment of the present application.
[0267] In actual application, the components in the electronic device 1500 are coupled together via the bus 1503. It is understood that the bus 1503 is used to realize the connection and communication between these components. In addition to the data bus, the bus 1503 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.15 Various buses are labeled as bus 1503.
[0268] The electronic device has the following beneficial effects:
[0269] By combining the method of comprehensive judgment of face and body, significant beneficial effects are achieved in the identification and positioning of occupants in the vehicle. Specifically, compared with a single face or body recognition method, this method has greatly improved the recall rate and accuracy. First, by combining the comprehensive information of face and body, the present application can more accurately identify occupants in a complex in-vehicle environment. Face information provides the facial features of the occupants, while body information provides more dimensional information such as the body shape and posture of the occupants. This fusion of multimodal information enables the system to overcome the limitations that may exist in a single information source when identifying occupants, such as failure of face recognition due to insufficient light, or inaccurate body recognition due to partial occupant body occlusion. Therefore, the comprehensive judgment method significantly improves the recall rate and accuracy of the system. Secondly, the present application uses a weighted bipartite graph and KM algorithm to match occupants and faces and bodies, further improving the matching accuracy. This method constructs a weighted adjacency matrix by calculating the distance or similarity between the headrest position and the face image and the key points of the human body. Then, the KM algorithm is used to find the optimal matching result on this adjacency matrix, that is, the sum of the weights of all edges that are successfully matched is minimized. This method not only takes into account the spatial position relationship between the occupant and the seat, but also takes into account other features between the occupant and the seat, thereby greatly improving the accuracy of matching. In addition, the present application uses the coordinate positioning of the key points of the human body to accurately determine where the occupant is sitting, rather than a simple regional determination. This method can accurately determine the specific position of the occupant and avoid the misjudgment that may be caused by regional determination. Finally, the present application can also give the child attributes of the occupant without increasing the time consumption. This is of great significance for vehicle safety monitoring and passenger management. By identifying child occupants, the system can automatically adjust the safety settings of the vehicle (such as the switch status of the airbag) to ensure the safety of child occupants. At the same time, the identification of child attributes can also provide valuable information for passenger management, such as the age distribution of passengers, the number of child passengers, etc. The realization of this function not only improves the intelligence level of the system, but also reduces resource consumption.
[0270] In summary, this application achieves significant beneficial effects in the identification and positioning of occupants in vehicles by combining the method of comprehensive determination of face and body, using weighted bipartite graph and KM algorithm for matching, determining the position of occupants by the coordinates of key points of the human body, and giving the child attributes of occupants. These beneficial effects not only improve the accuracy and reliability of the system, but also enhance the user experience and the safety of the vehicle.
[0271] The embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When the executable instructions are executed by at least one processor 1501, the method for detecting seating information of a person in a vehicle described in the embodiment of the present application is implemented.
[0272] In some embodiments, the storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface storage, an optical disk, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.
[0273] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0274] As an example, executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).
[0275] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.
[0276] The computer-readable storage medium has the following advantages:
[0277] The present application achieves significant beneficial effects in the identification and positioning of occupants in vehicles by combining the method of comprehensive determination of face and body. Specifically, the method has greatly improved the recall rate and accuracy compared with the single face or body recognition method. First, by combining the comprehensive information of face and body, the present application can more accurately identify occupants in a complex in-vehicle environment. Face information provides the facial features of the occupants, while body information provides more dimensional information such as the body shape and posture of the occupants. This fusion of multimodal information enables the system to overcome the limitations that may exist in a single information source when identifying occupants, such as failure of face recognition due to insufficient light, or inaccurate body recognition due to partial occupant body occlusion. Therefore, the comprehensive determination method significantly improves the recall rate and accuracy of the system. Secondly, the present application uses weighted bipartite graphs and KM algorithms to match occupants and faces and bodies, further improving the matching accuracy. This method constructs a weighted adjacency matrix by calculating the distance or similarity between the headrest position and the face image and the key points of the human body. Then, the KM algorithm is used to find the optimal matching result on this adjacency matrix, that is, the sum of the weights of all edges that are successfully matched is minimized. This method not only takes into account the spatial position relationship between the occupant and the seat, but also takes into account other features between the occupant and the seat, thereby greatly improving the accuracy of matching. In addition, the present application uses the coordinate positioning of the key points of the human body to accurately determine where the occupant is sitting, rather than a simple regional determination. This method can accurately determine the specific position of the occupant and avoid the misjudgment that may be caused by regional determination. Finally, the present application can also give the child attributes of the occupant without increasing the time consumption. This is of great significance for vehicle safety monitoring and passenger management. By identifying child occupants, the system can automatically adjust the safety settings of the vehicle (such as the switch status of the airbag) to ensure the safety of child occupants. At the same time, the identification of child attributes can also provide valuable information for passenger management, such as the age distribution of passengers, the number of child passengers, etc. The realization of this function not only improves the intelligence level of the system, but also reduces resource consumption.
[0278] In summary, this application achieves significant beneficial effects in the identification and positioning of occupants in vehicles by combining the method of comprehensive determination of face and body, using weighted bipartite graph and KM algorithm for matching, determining the position of occupants by the coordinates of key points of the human body, and giving the child attributes of occupants. These beneficial effects not only improve the accuracy and reliability of the system, but also enhance the user experience and the safety of the vehicle.
[0279] In the several embodiments provided in the present application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0280] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0281] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0282] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0283] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for detecting seat information of a person in a vehicle, characterized in that: The method comprises: Acquire an in-vehicle video image of a target vehicle, and perform target detection processing on the in-vehicle video image to obtain at least one face image and at least one body image; According to the in-car area information to which each human body image belongs, the human body image is cropped to obtain an optimized target human body image; wherein different in-car areas correspond to different cropping sizes; Performing human body key point recognition processing on the target human body image to obtain target joint point information and label information in the target human body image; wherein the label information indicates whether the target human body image is a child; Based on the label information in each of the target human images, the target joint point information in the target human image and the face center point in the at least one face image, determining the seating information of the person in the target vehicle; The determining of the seating information of the person in the target vehicle based on the label information in each of the target human images, the target joint point information in the target human image, and the face center point in the at least one face image includes: Determine a human body seating position based on target joint point information in the target human body image, and determine a human face seating position based on a human face center point in the at least one human face image; Determine the seating information of the person in the target vehicle based on the tag information, the seating position of the human body and the seating position of the face; The method for determining the human body seat position includes: determining the seat position information of each seat in the vehicle, and constructing a seat set based on the seat position information of each seat; wherein each element in the seat set represents the seat position information of a seat; constructing a joint point set based on the joint point information; wherein each element in the joint point set represents a joint point corresponding to a target human body image; performing a second matching process based on the seat set and the joint point set to obtain a first matching result, and using the first matching result as the human body seat position; The method for determining the seating position of the face includes: determining the headrest position information of each seat in the vehicle, and constructing a headrest set based on the headrest position information of each seat; wherein each element in the headrest set represents the headrest position information of a seat; constructing a face set based on each face image; wherein each element in the face set represents a face image; performing a third matching process based on the headrest set and the face set to obtain a second matching result, and using the second matching result as the seating position of the face.
2. The method according to claim 1, characterized in that The step of obtaining the in-vehicle video image of the target vehicle comprises: An in-vehicle video image of a target vehicle is collected by an image acquisition device; the image acquisition device is installed at an interior rearview mirror or a reading light inside the target vehicle, and the orientation of the image acquisition device forms an acute angle with the horizontal direction of the target vehicle.
3. The method according to claim 2, characterized in that The acute angle ranges from 20 degrees to 70 degrees.
4. The method according to claim 1, characterized in that: The performing target detection processing on the in-vehicle video image includes: Sampling the in-vehicle video image to obtain a sampled image; The sampling image is input into a pre-trained target sampling model to obtain the at least one face image and the at least one body image, each of the at least one face image or the at least one body image corresponds to a position information, and the position information is represented as the horizontal and vertical coordinates of the upper left corner and the lower right corner of the image frame including the image.
5. The seating information detection method according to claim 1, characterized in that: The method of performing image cropping processing on the human body image according to the in-vehicle area information to which each human body image belongs to obtain an optimized target human body image includes: Determining the cropping size of each human body image based on the in-vehicle area information to which each human body image belongs; wherein the in-vehicle area information includes a front row area and a rear row area, the cropping size includes a first cropping size and a second cropping size, the first cropping size of the human body images located in the front row area is the same, and the second cropping size of the human body images located in the rear row area is the same; Each human body image is cropped based on the cropping size of each human body image.
6. The method according to claim 5, characterized in that The in-vehicle area information to which each human body image belongs is determined in the following manner: For each of the human body images, determining a first intersection area between the human body image and the front row area and a second intersection area between the human body image and the back row area; Determine a first ratio of the first intersection area to the front area and a second ratio of the second intersection area to the back area; When the first ratio is greater than the second ratio, it is determined that the human body image is located in the front row, and when the second ratio is greater than the first ratio, it is determined that the human body image is located in the back row.
7. The method according to claim 5, characterized in that The determining the cropping size of each human image based on the in-vehicle region information to which each human image belongs includes: Determine a preset area corresponding to each front seat and each rear seat based on the seats in the vehicle; The cropping size is determined based on the preset area.
8. The method according to claim 7, characterized in that The step of performing cropping processing on each human body image based on the cropping size of each human body image comprises: Determine a third intersection area of each human body image and the preset area corresponding to the human body image; A cropping center point is determined based on the intersection area, and a cropping area of a specified size is determined according to the cropping center point, and each human body image is cropped based on the cropping area.
9. The method according to claim 1, characterized in that: The target joint point information in the target human body image is obtained by: The target human image is input into a human key point detection model to obtain the target joint point information; wherein the in-vehicle area information includes a front row area and a rear row area, and the human key point detection model uses different loss functions for the front row area and the rear row area during training.
10. The method according to claim 9, characterized in that The label information in the target human image is obtained by: The target human image is input into the human key point detection model to obtain the label information; wherein a judgment node is pre-added in the human key point detection model, and the judgment node judges whether the target human image is a child based on the detection result of the human key point detection model.
11. The method according to claim 1, characterized in that: Before determining the seating information of the person in the target vehicle, the method further includes: For the at least one face image and the at least one body image, respectively calculating the distance from each face image to each body image; Based on the calculation result, the at least one face image and the at least one body image are matched to obtain a face-body matching result; The method further comprises: The seating information of the person in the target vehicle is determined based on the face and body matching result; wherein the face and body matching result includes at least one of a successful face and body match, only a face, and only a body.
12. The method according to claim 11, characterized in that The step of calculating the distance from each face image to each body image respectively for the at least one face image and the at least one body image; The at least one face image and the at least one body image are matched based on the calculation result to obtain the face-body matching result, including: constructing a face set based on the at least one face image, and constructing a body set based on the at least one body image; Constructing a first adjacency matrix based on the face set and the body set; A first matching process is performed on the at least one face image and the at least one body image based on the first adjacency matrix to obtain the face-body matching result.
13. The method according to claim 12, characterized in that The performing a first matching process on the at least one face image and the at least one body image based on the first adjacency matrix to obtain the face-body matching result includes: Assigning a first top label to each element in the face set and assigning a second top label to each element in the body set; A complete match is obtained between each element in the face set and each element in the body set, wherein the sum of weights of all successfully matched edges is the smallest, and the weight of each edge is the distance between the first and second top marks of the two elements corresponding to the edge.
14. The method according to claim 1, characterized in that The performing a second matching process based on the seat set and the joint point set includes: Constructing a second adjacency matrix based on the seat set and the joint point set; Based on the second adjacency matrix, a complete match is obtained between each element in the seat set and each element in the joint point set, wherein the sum of weights of all successfully matched edges is the smallest, and the weight of each edge is the distance between the two elements corresponding to the edge.
15. The method according to claim 1, characterized in that The performing a third matching process based on the headrest set and the face set includes: Constructing a third adjacency matrix based on the headrest set and the face set; Based on the third adjacency matrix, a complete match is obtained between each element in the headrest set and each element in the face set, wherein the sum of weights of all successfully matched edges is the smallest, and the weight of each edge is the distance between the two elements corresponding to the edge.
16. A device for detecting seat information of a person in a vehicle, characterized in that: The device comprises: An acquisition module, used to acquire an in-vehicle video image of a target vehicle, and perform target detection processing on the in-vehicle video image to obtain at least one face image and at least one body image; A cropping module is used to crop the human body image according to the in-vehicle area information to which each human body image belongs, so as to obtain an optimized target human body image; wherein different in-vehicle areas correspond to different cropping sizes; A recognition module, used for performing human key point recognition processing on the target human image to obtain target joint point information and label information in the target human image; wherein the label information indicates whether the target human image is a child; A determination module, for determining the seating information of the personnel in the target vehicle based on the label information in each of the target human images, the target joint point information in the target human image and the face center point in the at least one face image; the determination of the seating information of the personnel in the target vehicle based on the label information in each of the target human images, the target joint point information in the target human image and the face center point in the at least one face image comprises: determining the human seat position based on the target joint point information in the target human image, and determining the face seat position based on the face center point in the at least one face image; determining the seating information of the personnel in the target vehicle based on the label information, the human seat position and the face seat position; the method for determining the human seat position comprises: determining the seat position information of each seat in the vehicle, and constructing a seat set based on the seat position information of each seat; wherein , each element in the seat set represents the seat position information of a seat; a joint point set is constructed based on the joint point information; wherein each element in the joint point set represents a joint point corresponding to a target human body image; a second matching process is performed based on the seat set and the joint point set to obtain a first matching result, and the first matching result is used as the human body sitting position; a method for determining the face sitting position includes: for each seat in the car, determining the headrest position information of each seat, and constructing a headrest set based on the headrest position information of each seat; wherein each element in the headrest set represents the headrest position information of a seat; a face set is constructed based on each face image; wherein each element in the face set represents a face image; a third matching process is performed based on the headrest set and the face set to obtain a second matching result, and the second matching result is used as the face sitting position.
17. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to execute the method for detecting seating information of a person in a vehicle as described in any one of claims 1 to 15.
18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting the seating information of a person in a vehicle as claimed in any one of claims 1 to 15 is executed.
