Method and device for detecting personnel in vehicle cabin, electronic equipment, medium and vehicle

Abstract

The invention relates to a method and device for detecting personnel in a vehicle cabin, electronic equipment, a medium and a vehicle, and particularly relates to the technical field of intelligent vehicle control. Comprising the steps that a plurality of image sets are acquired, and different image sets correspond to different camera azimuth information and / or camera type information; calculating a timestamp difference value between a target frame and an adjacent frame in the plurality of image sets; wherein the target frame and the adjacent frame are adjacent in time sequence and belong to different image sets; under the condition that the timestamp difference values are all within a preset threshold range, acquiring adjacent images of a preset frame number from the plurality of image sets; and inputting the preset frame number of adjacent images into the preset visual algorithm model to obtain the position of the personnel in the cabin output by the preset visual algorithm model, thereby improving the detection accuracy of the personnel in the vehicle cabin.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent vehicle control technology, and in particular to a method, device, electronic equipment, medium, and vehicle for detecting a person in a vehicle cabin. Background Art

[0002] Currently, vehicles have at least two cameras positioned in different locations within the cabin, such as a front-row camera and a rear-row camera. Images captured by these cameras can be used to detect the location of occupants within the cabin, fulfilling the Occupant Monitoring System (OMS) service and enabling intelligent spatial interaction within the vehicle. However, the different types and locations of cameras produce images at varying frame rates, creating the risk of frame rate deviation. Consequently, the accuracy of occupant location detection using these images is low. Summary of the Invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, electronic equipment, medium and vehicle for detecting people in the vehicle cabin, which can avoid the risk of frame rate deviation and improve the accuracy of occupant position detection in the vehicle cabin.

[0004] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0005] In a first aspect, the present disclosure provides a method for detecting a person in a vehicle cabin, comprising:

[0006] Acquire multiple image sets, wherein different image sets correspond to different camera orientation information and / or different camera type information; and the image frames included in the multiple image sets are arranged in time sequence;

[0007] Calculating a timestamp difference between a target frame and adjacent frames in a plurality of image sets, wherein the target frame and the adjacent frames are adjacent in time sequence and belong to different image sets;

[0008] When the timestamp differences are all within a preset threshold range, obtaining a preset number of adjacent images from the plurality of image sets;

[0009] A preset number of adjacent images are input into a preset visual algorithm model to obtain the position of people in the cabin output by the preset visual algorithm model. The preset visual algorithm model is a pre-trained model for identifying the position of people in images.

[0010] As an optional implementation of the embodiment of the present disclosure, the preset threshold range is a time range from a first preset threshold to a second preset threshold, and the first preset threshold is less than the second preset threshold;

[0011] After calculating the timestamp difference between the target frame and adjacent frames in the multiple image sets, it also includes:

[0012] When the timestamp difference is less than a first preset threshold, deleting the target frame from the first image set corresponding to the target frame to update the first image set;

[0013] When the timestamp difference is greater than a second preset threshold, the adjacent frames are deleted from the second image set corresponding to the adjacent frames to update the second image set.

[0014] As an optional implementation of the embodiment of the present disclosure, after deleting adjacent frames from the second image set to update the second image set when the timestamp difference is greater than a second preset threshold, and before acquiring a preset number of adjacent frames from multiple image sets when the timestamp differences are all within a preset threshold range, the method further includes:

[0015] For the updated first image set and the updated second image set, calculating an update timestamp difference between a pending frame and a pending adjacent frame until the update timestamp difference is within a preset threshold range; wherein the pending frame is any frame in the updated first image set;

[0016] The timestamp differences corresponding to other image sets in the plurality of image sets are calculated, where the other image sets are image sets other than the first image set and the second image set in the plurality of image sets.

[0017] As an optional implementation of the embodiment of the present disclosure, after acquiring multiple image sets and before calculating the timestamp difference values ​​between the target frame and adjacent frames in the multiple image sets, the method further includes:

[0018] sorting all image frames included in different image sets to which the target frame and adjacent frames belong according to the timestamp sizes of the image frames;

[0019] All sorted image frames are stored in a queue, where the target frame and adjacent frames are adjacent in sequence.

[0020] As an optional implementation of the embodiment of the present disclosure, the training process of the preset visual algorithm model includes:

[0021] Obtaining sample image sets corresponding to different camera orientation information and / or camera type information, and person location labels corresponding to the sample image sets;

[0022] Input the sample image set into the initial visual algorithm model to obtain the predicted personnel position output by the initial visual algorithm model;

[0023] Based on the preset loss function, the loss value between the predicted person position and the person position label corresponding to the sample image set is calculated;

[0024] When the loss value is greater than the loss threshold, the model parameters of the initial visual algorithm model are adjusted until the loss value is less than or equal to the loss threshold, thereby obtaining a converged preset visual algorithm model.

[0025] As an optional implementation of the embodiment of the present disclosure, the camera orientation information includes front row and rear row; the camera type information includes infrared and color;

[0026] The multiple image sets include at least two of the image sets corresponding to the front-row infrared camera, the image sets corresponding to the front-row color camera, the image sets corresponding to the rear-row infrared camera, and the image sets corresponding to the rear-row color camera.

[0027] In a second aspect, the present disclosure provides a device for detecting a person in a vehicle cabin, the device comprising:

[0028] An acquisition module is configured to acquire a plurality of image sets, wherein different image sets correspond to different camera orientation information and / or different camera type information; and the image frames included in the plurality of image sets are arranged in a time sequence;

[0029] A calculation module, configured to calculate a timestamp difference between a target frame and adjacent frames in a plurality of image sets; wherein the target frame and the adjacent frames are adjacent in time sequence and belong to different image sets;

[0030] The acquisition module is further configured to acquire a preset number of adjacent images from the plurality of image sets when the timestamp differences are all within a preset threshold range;

[0031] The recognition module is used to input adjacent images of a preset number of frames into a preset visual algorithm model to obtain the position of people in the cabin output by the preset visual algorithm model. The preset visual algorithm model is a pre-trained model for identifying the position of people in the image.

[0032] As an optional implementation of the embodiment of the present disclosure, the preset threshold range is a time range from a first preset threshold to a second preset threshold, and the first preset threshold is less than the second preset threshold;

[0033] The calculation module is further configured to: when the timestamp difference is less than a first preset threshold, delete the target frame from the first image set corresponding to the target frame to update the first image set;

[0034] When the timestamp difference is greater than a second preset threshold, the adjacent frames are deleted from the second image set corresponding to the adjacent frames to update the second image set.

[0035] As an optional implementation of the embodiment of the present disclosure, the calculation module is further configured to: calculate, for the updated first image set and the updated second image set, an update timestamp difference between a pending frame and a pending adjacent frame until the update timestamp difference falls within a preset threshold range; wherein the pending frame is any frame in the updated first image set;

[0036] The timestamp differences corresponding to other image sets in the plurality of image sets are calculated, where the other image sets are image sets other than the first image set and the second image set in the plurality of image sets.

[0037] As an optional implementation of the embodiment of the present disclosure, the acquisition module, after acquiring the multiple image sets, is further configured to: sort all image frames included in different image sets to which the target frame and adjacent frames belong according to timestamp sizes of the image frames;

[0038] All sorted image frames are stored in a queue, where the target frame and adjacent frames are adjacent in sequence.

[0039] As an optional implementation of the embodiment of the present disclosure, the device for detecting people in a vehicle cabin further includes: a training module, configured to: train a preset visual algorithm model, including: obtaining sample image sets corresponding to different camera orientation information and / or camera type information, and person location labels corresponding to the sample image sets;

[0040] Input the sample image set into the initial visual algorithm model to obtain the predicted personnel position output by the initial visual algorithm model;

[0041] Based on the preset loss function, the loss value between the predicted person position and the person position label corresponding to the sample image set is calculated;

[0042] When the loss value is greater than the loss threshold, the model parameters of the initial visual algorithm model are adjusted until the loss value is less than or equal to the loss threshold, thereby obtaining a converged preset visual algorithm model.

[0043] As an optional implementation of the embodiment of the present disclosure, the camera orientation information includes front row and rear row; the camera type information includes infrared and color;

[0044] The multiple image sets include at least two of the image sets corresponding to the front-row infrared camera, the image sets corresponding to the front-row color camera, the image sets corresponding to the rear-row infrared camera, and the image sets corresponding to the rear-row color camera.

[0045] In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for detecting a person in a vehicle cabin as described in the first aspect or any one of its optional embodiments is implemented.

[0046] In a fourth aspect, the present disclosure provides a computer-readable storage medium, comprising: a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the method for detecting people in the vehicle cabin as described in the first aspect or any one of its optional embodiments.

[0047] In a fifth aspect, the present disclosure provides a vehicle comprising: a plurality of cameras of different orientations and / or types, and a device for detecting people in a vehicle cabin as described in the second aspect or any optional embodiment thereof, or an electronic device as described in the third aspect.

[0048] In a sixth aspect, the present disclosure provides a computer program product, comprising: the computer program product comprises a computer program, and when the computer program runs on a computer, the computer implements the method for detecting people in a vehicle cabin as described in the first aspect or any one of its optional embodiments.

[0049] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0050] The present disclosure provides a method, apparatus, electronic device, medium, and vehicle for detecting people in a vehicle cabin. The method first acquires multiple image sets, each corresponding to different camera orientation information and / or camera type information, and the images included in the multiple image sets are arranged in a time sequence. The method then calculates the timestamp difference between a target frame and adjacent frames in the multiple image sets, where the target frame and adjacent frames are temporally adjacent and belong to different image sets. Furthermore, when the timestamp difference is within a preset threshold range, a preset number of adjacent frames are acquired from the multiple image sets, and the preset number of adjacent frames are input into a pre-trained preset visual algorithm model for identifying the position of a person in an image, thereby obtaining the position of the person in the cabin output by the preset visual algorithm model. In this way, the images captured by different cameras can be aligned in time, and the images processed after alignment have no frame rate deviation. These images are then input into the preset visual algorithm model for recognition, thereby avoiding the risk of frame rate deviation in images captured by different cameras affecting the OMS system service, thereby making the detection of the position of the person in the vehicle cabin more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0052] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 1 is a flow chart of a method for detecting a person in a vehicle cabin according to an embodiment of the present disclosure;

[0054] Figure 2 A schematic structural diagram of a device for detecting a person in a vehicle cabin provided by an embodiment of the present disclosure;

[0055] Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0058] To address some or all of the technical problems in the related art, the present disclosure provides a method, apparatus, electronic device, medium, and vehicle for detecting a person in a vehicle cabin. The method first acquires multiple image sets, each corresponding to different camera orientation information and / or camera type information, and each including image frames arranged in a time sequence. The method then calculates the timestamp difference between a target frame and adjacent frames in the multiple image sets, wherein the target frame and adjacent frames are temporally adjacent and belong to different image sets. Furthermore, when the timestamp difference is within a preset threshold, a preset number of adjacent frames are acquired from the multiple image sets and input into a pre-trained visual algorithm model for identifying the position of a person in an image to obtain the position of the person in the cabin output by the pre-trained visual algorithm model. In this way, the present disclosure can align images captured by different cameras in time. The aligned images have no frame rate deviation. These images are input into the pre-trained visual algorithm model for recognition, thereby avoiding the risk of frame rate deviation in images captured by different cameras that may affect the OMS service, thereby making the detection of the position of the person in the vehicle cabin more accurate.

[0059] A method for detecting people in a vehicle cabin provided in an embodiment of the present disclosure can be implemented by a detection device or electronic device for people in the vehicle cabin, and the electronic device includes but is not limited to a vehicle terminal, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiment of the present disclosure does not limit this. The electronic device can be run alone to implement the present disclosure, or it can be connected to a network and implement the present disclosure through interactive operations with other computer devices in the network. Among them, the network where the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.

[0060] It should be noted that the protection scope of the method for detecting people in a vehicle cabin described in the embodiment of the present disclosure is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the existing technology based on the principles of the present disclosure are included in the protection scope of the present disclosure.

[0061] like Figure 1 As shown, Figure 1 This is a flow chart of a method for detecting a person in a vehicle cabin according to an embodiment of the present disclosure. This method can be performed by a device for detecting a person in a vehicle cabin, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device, such as a vehicle terminal. Figure 1 As shown, the method mainly includes the following steps S101 to S104:

[0062] S101. Acquire multiple image sets.

[0063] The image frames included in the multiple image sets are arranged in a time sequence. Different image sets correspond to different camera orientation information and / or different camera type information. Camera orientation information includes, but is not limited to, front row and rear row, and camera type information includes, but is not limited to, infrared (IR) and color (RGB), etc., which are not specifically limited in this disclosure.

[0064] The method for detecting a person in a vehicle cabin provided by the embodiments of the present disclosure can be applied to a vehicle terminal. The vehicle terminal can control cameras corresponding to different orientation information and / or different types of information to capture images, thereby obtaining multiple image sets. It is understood that cameras of different orientations and / or different types of information can be controlled to capture images.

[0065] In some embodiments, during the acquisition of multiple image sets, the same camera corresponding to the same camera orientation information and / or camera type information is first controlled to capture images, resulting in multiple original image sets. Each original image set corresponds to the same camera orientation information and / or camera type information, with each original image corresponding to each camera. For each of these multiple original image sets, the preceding and following adjacent image frames are compared to determine whether they meet a fusion condition. If so, the preceding and following adjacent image frames are fused to obtain a new image. It should be noted that the timestamp corresponding to the new image is the same as the timestamp of the later of the two consecutive frames.

[0066] It can be understood that the images captured by the same camera are used as an original image set, and different cameras are controlled to capture images to obtain multiple original image sets. Then, any two adjacent frames in each original image set are judged to determine whether they meet the fusion condition. If they do, the two adjacent frames are fused to obtain a new image.

[0067] The above embodiment improves image quality by fusing previous and next frame images captured by the same camera, thereby obtaining multiple high-quality image sets.

[0068] Acquire multiple image sets, each of which is captured by cameras of different orientations and / or types. The multiple image sets include, but are not limited to, at least two of the following: an image set corresponding to a front-row infrared (IR) camera, an image set corresponding to a front-row color (RGB) camera, an image set corresponding to a rear-row infrared (IR) camera, and an image set corresponding to a rear-row color (RGB) camera.

[0069] For example, the camera orientation information corresponding to the image set IR_front is the front row, and the corresponding camera type information is IR, indicating that the image set IR_front is captured by the front row IR camera; the camera orientation information corresponding to the image set RGB_behind is the back row, and the corresponding camera type information is RGB, indicating that the image set RGB_behind is captured by the back row RGB image.

[0070] In some embodiments, images captured by cameras of different orientations and / or different types are arranged in time sequence to obtain multiple image sets. It can be understood that all image frames are sorted according to the corresponding timestamp size, and each image frame in the sequence carries the image set identifier to which it belongs to distinguish which orientation or type of camera the image frame was captured by.

[0071] S102: Calculate the timestamp difference between the target frame and adjacent frames in the plurality of image sets.

[0072] The target frame and the adjacent frame are temporally adjacent and belong to different image sets. Optionally, the target frame belongs to a first image set and the adjacent frame belongs to a second image set. The first image set and the second image set are any two image sets from a plurality of image sets. The target frame is any frame in the first image set. Since the image frames are arranged in temporal sequence, adjacent frames that are temporally adjacent to the target frame and belong to another image set (the second image set) can be determined.

[0073] In some embodiments, the camera orientation information corresponding to the first image set and the second image set may be the same, but the camera type information is different, or the camera type information corresponding to the first image set and the second image set is the same, but the camera orientation information is different, or both the camera orientation information and the camera type information corresponding to the first image set and the second image set are different.

[0074] Exemplarily, the first image set is the image set IR_front, and the second image set is the image set RGB_front; the first image set is the image set IR_behind, and the second image set is the image set RGB_behind; the first image set is the image set IR_front, and the second image set is the image set IR_behind.

[0075] The target frame is any frame in the first image set, for example, the first frame in the first image set. Each target frame has a timestamp that indicates when it was generated. For example, if target frame IR_f1 is the first frame in the first image set IR_front, the timestamp t1 indicates that the front-row IR camera generated target frame IR_f1 at time t1.

[0076] An adjacent frame is an image frame in the second image set that is adjacent to the image frame corresponding to the target frame in image order, for example, the second frame in the second image set. It is understood that an adjacent frame can be an image frame that precedes or follows the target frame in chronological order. Each adjacent frame has a timestamp that indicates the time when the adjacent frame was generated.

[0077] In some embodiments, after executing step S101 (obtaining multiple image sets) and before executing step S102 (calculating the timestamp difference between the target frame and the adjacent frames in the multiple image sets), all image frames included in the first image set and the second image set are first sorted according to the timestamp size of the image frames, and all the sorted image frames are stored in a queue in sequence, in which the target frame and the adjacent frames are adjacent in sequence.

[0078] It is understandable that after acquiring multiple image sets, two image sets are randomly selected: the first image set and the second image set, and all the image frames included in them are sorted according to the timestamp sizes corresponding to these image frames. All the sorted image frames are stored in a queue to facilitate the subsequent calculation of timestamp differences. In this way, the image frames in the queue are arranged in sequence according to the timestamp size, that is, according to the chronological order. The image frames of the first image set and the image frames of the second image set may be interspersed and mixed with each other, and the image frames of the first image set may be preceded and followed by the image frames of the second image set. At this time, the target frame and the adjacent frames are sequentially adjacent in the queue. This facilitates the calculation of the timestamp difference between the target frame and the adjacent frames.

[0079] When calculating the timestamp difference between the target frame in the first image set and the adjacent frame in the second image set, exemplarily, the first image set IR_front, the second image set RGB_front, the target frame IR_fi, the adjacent frame RGB_f(i+1), i is an integer greater than or equal to 1; wherein the timestamp t of the target frame IR_fi is i , the timestamp t of the adjacent frame RGB_f(i+1) i+1 , calculate the timestamp difference Δt between the target frame IR_fi and the adjacent frame RGB_f(i+1) f =t i -t i+1 .

[0080] For example, the first image set IR_behind, the second image set RGB_behind, the target frame IR_bj, the adjacent frame RGB_b(j-1), j is an integer greater than or equal to 2; wherein the timestamp t of the target frame IR_bj is j , the timestamp t of the adjacent frame RGB_b(j-1) j+1 , calculate the timestamp difference Δt between the target frame IR_bj and the adjacent frame RGB_b(j-1) b =t j -t j-1 .

[0081] For example, the first image set IR_front, the second image set IR_behind, the target frame IR_fi, the adjacent frame IR_b(i+1), i is an integer greater than or equal to 1; wherein the timestamp t of the target frame IR_fi is i , the timestamp t of the adjacent frame IR_b(i+1) b(i+1) , calculate the timestamp difference Δt=t between the target frame IR_fi and the adjacent frame IR_b(i+1) i -t b(i+1) .

[0082] After calculating the timestamp difference between the target frame and adjacent frames in the plurality of image sets, the timestamp difference is compared with a preset threshold range to filter out the image frames that are aligned in time.

[0083] The preset threshold range is a preset time range. When the time stamp difference between two adjacent frames of images is within the preset threshold range, it can be considered that the two adjacent frames of images are aligned in time.

[0084] In some embodiments, the preset threshold range is a time range from a first preset threshold to a second preset threshold, where the first preset threshold is less than the second preset threshold. The preset threshold range can be expressed as [a, b], indicating that the timestamp difference should be greater than or equal to a and less than or equal to b. For example, a = -33 ms, b = 33 ms. This disclosure does not specifically limit the size of the preset threshold range; this is provided for illustrative purposes only.

[0085] If the timestamp difference between the target frame in the first image set and the adjacent frames in the second image set is less than a first preset threshold, the target frame is deleted from the first image set to obtain an updated first image set. Specifically, if the timestamp difference between the target frame in the first image set and the adjacent frames in the second image set is less than the first preset threshold, indicating that the target frame was generated too early and is difficult to align with the adjacent frames in time, the target frame is deleted from the first image set to obtain an updated first image set.

[0086] For example, following the previous example, the timestamp difference Δt between the target frame IR_fi and the adjacent frame RGB_f(i+1) is calculated. f =t i -t i+1 , Δt f <a, indicating that the target frame IR_fi is generated at least 33ms earlier than the adjacent frame RGB_f(i+1), and the target frame IR_fi is deleted from the first image set IR_front.

[0087] If the timestamp difference between the target frame in the first image set and the adjacent frame in the second image set is greater than a second preset threshold, the adjacent frame is deleted from the second image set to obtain an updated second image set. Specifically, if the timestamp difference between the target frame in the first image set and the adjacent frame in the second image set is greater than the second preset threshold, indicating that the adjacent frame was generated too early compared to the target frame and is difficult to align with the target frame in time, the adjacent frame is deleted from the second image set to obtain an updated second image set.

[0088] For example, following the previous example, the timestamp difference Δt between the target frame IR_fi and the adjacent frame RGB_f(i+1) is calculated. f =t i -t i+1, Δt f >b, indicating that the target frame IR_fi is generated at least 33ms later than the adjacent frame RGB_f(i+1), and the adjacent frame RGB_f(i+1) is deleted from the second image set RGB_front.

[0089] In the above embodiment, the time sequence of two adjacent frames of images in any two different image sets among multiple image sets is compared. If the time stamp difference between the two adjacent frames of images is not within a preset threshold range, the newer image that is later in time is retained to obtain an updated image set.

[0090] In some embodiments, based on the above embodiments, after the first and second image sets are updated, the updated timestamp difference between the pending frame in the updated first image set and the pending adjacent frame in the updated second image set is calculated until the updated timestamp difference falls within a preset threshold range, and then the timestamp differences corresponding to other image sets in the multiple image sets are calculated. The pending frame is any frame in the updated first image set, and the pending adjacent frame is an image frame in the updated second image set that is adjacent in image sequence to the image frame corresponding to the pending frame.

[0091] In the above embodiment, image frames of any two image sets in multiple image sets are screened according to a preset threshold range, and image frames whose inter-frame timestamp difference is within the preset threshold range are retained. When the inter-frame timestamp difference is less than the first preset threshold or greater than the second preset threshold, the image frames with smaller timestamps are deleted, and the image set is updated to complete one round of judgment and screening, and then the next round of judgment and screening is performed until the timestamp difference corresponding to any two image sets meets the preset threshold range, and then another two image sets are replaced for image frame screening, until all image sets are traversed and the screening of all image frames in the image set is completed, so that all image frames are synchronized or approximately synchronized in time.

[0092] S103: when the timestamp differences are all within a preset threshold range, acquiring a preset number of frames of images from the multiple image sets;

[0093] The preset number of frames is a preset number of frames for acquiring images from the image set, and the preset number of frames may be 4.

[0094] In some embodiments, when the timestamp difference is within a preset threshold range, any adjacent images in a plurality of image sets can be considered to be synchronized in time, and then a preset number of frames of images are obtained from the plurality of images, and any adjacent images in the preset number of frames of images are also synchronized in time.

[0095] Exemplarily, image IR_f1 is obtained from the image set IR_front, image RGB_f2 is obtained from the image set RGB_front, image IR_b3 is obtained from the image set IR_behind, and image RGB_b4 is obtained from the image set RGB_behind, to obtain four adjacent images: image IR_f1, image RGB_f2, image IR_b3, and image RGB_b4.

[0096] S104: Input the preset number of frames of images into a preset visual algorithm model to obtain the number of people in the cabin output by the preset visual algorithm model.

[0097] Among them, the preset algorithm model is a pre-trained model used to identify the location of people in images.

[0098] In some embodiments, the training process of the preset visual algorithm model includes: first obtaining a sample image set corresponding to different camera orientation information and / or camera type information, and a personnel position label corresponding to the sample image set, and then inputting the sample image set into the initial visual algorithm model to obtain the predicted personnel position output by the initial visual algorithm model, and based on the preset loss function, calculating the loss value between the predicted personnel position and the personnel position label corresponding to the sample data set, and then, when the loss value is greater than the loss threshold, adjusting the model parameters of the initial visual algorithm model until the loss value is less than or equal to the loss threshold, thereby obtaining a converged preset visual algorithm model.

[0099] Among them, the initial visual algorithm model is constructed based on a deep learning model. The specific model structure can be referred to the prior art and will not be described in detail in this disclosure. The preset loss function can be a cross entropy loss function, etc., which is not specifically limited in this disclosure.

[0100] In the process of obtaining the sample image set and the corresponding person location labels, each sample image in the sample image set is marked with a person location label through manual annotation or machine learning. The person location label indicates the position coordinates of the human body area in the image, which is used as the true value.

[0101] Afterwards, the sample image set is used as the input of the initial visual algorithm model, and the initial visual algorithm model processes each sample image in the sample image set to output the predicted personnel position, which represents the position coordinates of the human body area in each sample image predicted by the model. This is used as the predicted value. Based on the preset loss function, the loss value between the predicted value and the true value is calculated to represent the gap between the predicted value and the true value. The loss value is compared with the loss threshold. If the loss value is greater than the preset threshold, it means that there is a large gap between the predicted value and the true value, and the model has not converged and further training is required. Then, the model parameters of the initial visual algorithm model are adjusted, and the loss value is calculated again until the loss value is less than or equal to the preset threshold, indicating that the model converges, and a converged preset visual algorithm model is obtained. This converged preset visual algorithm model can accurately identify the position of people in the cabin.

[0102] A preset number of frames of images are input into a preset visual algorithm model, and the preset visual algorithm model outputs the occupant location within the cabin. The preset number of frames of images are temporally aligned, with adjacent images captured by cameras of different orientations and / or types. This avoids the risk of frame rate deviations between images captured by different cameras, making the preset visual algorithm model's recognition and detection more accurate, thereby improving the accuracy of occupant location detection within the cabin.

[0103] In summary, the embodiment of the present disclosure provides a method for detecting people in a vehicle cabin. The method first obtains multiple image sets, and the camera orientation information and / or camera type information corresponding to the multiple image sets are different. The image frames included in the multiple image sets are arranged in time sequence; then the timestamp difference between the target frame and the adjacent frames in the multiple image sets is calculated, and the target frame and the adjacent frames belong to different image sets; then, when the timestamp difference is within a preset threshold range, a preset number of adjacent images are obtained from the multiple image sets, and the preset number of adjacent images are input into a pre-trained preset visual algorithm model for identifying the position of people in the image to obtain the position of people in the cabin output by the preset visual algorithm model. In this way, the embodiment of the present disclosure can align the images captured by different cameras in time. The images after alignment processing have no frame rate deviation. This part of the images are input into the preset visual algorithm model for recognition, which can avoid the risk of frame rate deviation of images captured by different cameras affecting the OMS system service, making the detection of the position of people in the vehicle cabin more accurate.

[0104] In one possible scenario, a vehicle cabin is provided with: a front IR camera, a front RGB camera, a rear IR camera, and a rear RGB camera. In this possible scenario, an embodiment of the present disclosure provides a method for detecting a person in the cabin, including the following steps S01 to S15:

[0105] S01. Obtain an image set IR_front captured by a front IR camera, an image set RGB_front captured by a front RGB camera, an image set IR_behind captured by a rear IR camera, and an image set RGB_behind captured by a rear RGB camera.

[0106] S02. Calculate the timestamp difference Δt between the target frame in the image set IR_front and the adjacent frames in the image set RGB_front. f .

[0107] For example, the target frame IR_fi and the adjacent frame RGB_f(i+1), i is an integer greater than or equal to 1; wherein the timestamp t of the target frame IR_fi is i , the timestamp t of the adjacent frame RGB_f(i+1) i+1 , calculate the timestamp difference Δt between the target frame IR_fi and the adjacent frame RGB_f(i+1) f =t i -t i+1 .

[0108] S03, determine the timestamp difference Δt f Whether it is within the preset threshold range [a,b].

[0109] The preset threshold range is a time range from a first preset threshold a to a second preset threshold b, indicating that the timestamp difference should be greater than or equal to the first preset threshold a and less than or equal to the second preset threshold b. For example, the preset threshold range is [-33, 33].

[0110] If not, the timestamp difference Δt f When the timestamp difference Δt is less than the first preset threshold a, S04 is executed to delete the target frame IR_fi from the image set IR_front; f If the timestamp difference is greater than the second preset threshold b, step S05 is executed to delete the adjacent frame RGB_f(i+1) from the image set RGB_front. It should be noted that after executing step S04 or step S05, the image set is updated, and the process returns to step S02 to reselect a target frame and calculate the timestamp difference until the timestamp difference falls within the preset threshold range, and then step S06 is executed.

[0111] If yes, it means that the images captured by the front IR camera and the front RGB camera are synchronized in time, avoiding the risk of frame rate deviation of all front cameras, and then executing step S06, calculating the timestamp difference Δt between the target frame in the image set IR_behind and the adjacent frames in the image set RGB_behind b .

[0112] For example, the target frame IR_bj and the adjacent frame RGB_b(j+1) are integers greater than or equal to 1; wherein the timestamp t of the target frame IR_bj is j , the timestamp t of the adjacent frame RGB_b(j+1) j+1 , calculate the timestamp difference Δt between the target frame IR_bj and the adjacent frame RGB_b(j+1) b =t j -t j+1 .

[0113] S07, determine the timestamp difference Δt b Whether it is within the preset threshold range [a,b].

[0114] If not, the timestamp difference Δt b When the timestamp difference Δt is less than the first preset threshold a, S08 is executed to delete the target frame IR_bj from the image set IR_behind; b If the timestamp difference is greater than the second preset threshold value b, step S09 is executed to delete the adjacent frame RGB_b(j+1) from the image set RGB_behind. It should be noted that after executing step S08 or step S09, the image set is updated, and the process returns to step S06 to reselect a target frame and calculate the timestamp difference until the timestamp difference falls within the preset threshold range, and then step S10 is executed.

[0115] If yes, it means that the images captured by the rear IR camera and the rear RGB camera are synchronized in time, avoiding the risk of frame rate deviation of all rear cameras, and then executing step S10, calculating the timestamp difference Δt between the target frame in the image set IR_front and the adjacent frames in the image set IR_behind I .

[0116] Exemplarily, the target frame IR_fi, the adjacent frame IR_b(i+1), i is an integer greater than or equal to 1; wherein the timestamp t of the target frame IR_fi i , the timestamp t of the adjacent frame IR_b(i+1) b(i+1) , calculate the timestamp difference Δt between the target frame IR_fi and the adjacent frame IR_b(i+1) I =t i -t b(i+1) .

[0117] S11. Determine the timestamp difference Δt I Whether it is within the preset threshold range [a,b].

[0118] If not, the timestamp difference Δt IIf the timestamp difference Δt is less than the first preset threshold a, execute S12, delete the target frame IR_fi from the image set IR_front; I If the timestamp difference is greater than the second preset threshold value b, step S13 is executed to delete the adjacent frame IR_b(i+1) from the image set IR_behind. It should be noted that after executing step S12 or step S13, the image set is updated, and the process returns to step S10 to reselect a target frame and calculate the timestamp difference until the timestamp difference is within the preset threshold range, and then step S14 is executed.

[0119] If so, it means that the images captured by the front IR camera and the rear IR camera are synchronized in time, avoiding the risk of frame rate deviation that may exist in all IR cameras, and then executing step S14, obtaining adjacent images of a preset number of frames from the image set IR_front, image set RGB_front, image set IR_behind and image set RGB_behind.

[0120] Exemplarily, one frame of image is obtained from the image set IR_front, the image set RGB_front, the image set IR_behind, and the image set RGB_behind, respectively, including: image IR_f1, image RGB_f2, image IR_b3, and image RGB_b4, and these four frames of image are obtained to be adjacent.

[0121] S15. Inputting a preset number of adjacent images into a preset visual algorithm model to obtain the position of the person in the cabin output by the preset visual algorithm model.

[0122] The above steps S11 may also include the calculation and judgment of the corresponding timestamp difference between the image set RGB_front and the image set RGB_behind, so that the images captured by the RGB camera are synchronized in time and the frame rate deviation that may exist in the RGB camera is avoided. The specific implementation is similar to the above embodiment and will not be elaborated in this disclosure.

[0123] In summary, in a scenario where a front IR camera, a front RGB camera, a rear IR camera, and a rear RGB camera are installed in a vehicle cabin, the images captured by each camera are temporally aligned to accurately implement the OMS service using temporally aligned adjacent images of a preset number of frames, thereby improving the detection accuracy of people in the cabin.

[0124] like Figure 2 As shown, Figure 2 This is a schematic structural diagram of a device for detecting a person in a vehicle cabin provided by an embodiment of the present disclosure, the device comprising:

[0125] An acquisition module 201 is configured to acquire a plurality of image sets, wherein different image sets correspond to different camera orientation information and / or different camera type information; and the image frames included in the plurality of image sets are arranged in a time sequence;

[0126] A calculation module 202 is configured to calculate a timestamp difference between a target frame and adjacent frames in a plurality of image sets, wherein the target frame and the adjacent frames are adjacent in time sequence and belong to different image sets;

[0127] The acquisition module 201 is further configured to acquire a preset number of adjacent images from the plurality of image sets when the timestamp differences are all within a preset threshold range;

[0128] The recognition module 203 is used to input adjacent images of a preset number of frames into a preset visual algorithm model to obtain the position of the personnel in the cabin output by the preset visual algorithm model. The preset visual algorithm model is a pre-trained model for identifying the position of personnel in images.

[0129] As an optional implementation of the embodiment of the present disclosure, the preset threshold range is a time range from a first preset threshold to a second preset threshold, and the first preset threshold is less than the second preset threshold;

[0130] The calculation module 202 is further configured to: when the timestamp difference is less than a first preset threshold, delete the target frame from the first image set corresponding to the target frame to update the first image set;

[0131] When the timestamp difference is greater than a second preset threshold, the adjacent frames are deleted from the second image set corresponding to the adjacent frames to update the second image set.

[0132] As an optional implementation of the embodiment of the present disclosure, the calculation module 202 is further configured to: calculate, for the updated first image set and the updated second image set, an update timestamp difference between a pending frame and a pending adjacent frame until the update timestamp difference falls within a preset threshold range; wherein the pending frame is any frame in the updated first image set;

[0133] The timestamp differences corresponding to other image sets in the plurality of image sets are calculated, where the other image sets are image sets other than the first image set and the second image set in the plurality of image sets.

[0134] As an optional implementation of the embodiment of the present disclosure, the acquisition module 201, after acquiring the multiple image sets, is further configured to: sort all image frames included in the first image set and the second image set according to the timestamp size of the image frames;

[0135] All sorted image frames are stored in a queue, where the target frame and adjacent frames are adjacent in sequence.

[0136] As an optional implementation of the embodiment of the present disclosure, the device for detecting people in a vehicle cabin further includes: a training module, configured to: train a preset visual algorithm model, including: obtaining sample image sets corresponding to different camera orientation information and / or camera type information, and person location labels corresponding to the sample image sets;

[0137] Input the sample image set into the initial visual algorithm model to obtain the predicted personnel position output by the initial visual algorithm model;

[0138] Based on the preset loss function, the loss value between the predicted person position and the person position label corresponding to the sample image set is calculated;

[0139] When the loss value is greater than the loss threshold, the model parameters of the initial visual algorithm model are adjusted until the loss value is less than or equal to the loss threshold, thereby obtaining a converged preset visual algorithm model.

[0140] As an optional implementation of the embodiment of the present disclosure, the camera orientation information includes front row and rear row; the camera type information includes infrared and color;

[0141] The multiple image sets include at least two of the image sets corresponding to the front-row infrared camera, the image sets corresponding to the front-row color camera, the image sets corresponding to the rear-row infrared camera, and the image sets corresponding to the rear-row color camera.

[0142] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. The electronic device includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When executed by the processor 301, the computer program implements the various processes of the method for detecting a person in a vehicle cabin in the above-mentioned method embodiment. The same technical effects can be achieved, and to avoid repetition, the details are not repeated here.

[0143] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the method for detecting people in a vehicle cabin in the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0144] The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0145] An embodiment of the present disclosure provides a vehicle, which includes: multiple cameras of different orientations and / or types, and a detection device for people in the vehicle cabin as in the above embodiment, or an electronic device as in the above embodiment, which can implement each process of the method for detecting people in the vehicle cabin in the above method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described here.

[0146] An embodiment of the present disclosure provides a computer program product, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the method for detecting people in the vehicle cabin in the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0147] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0148] In the several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0149] In the present disclosure, a processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0150] In this disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0151] In this disclosure, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0152] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0153] The above are merely specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to these embodiments, but is to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting people in a vehicle cabin, characterized in that: include: Acquire multiple image sets, wherein different image sets correspond to different camera orientation information and / or camera type information; the image frames included in the multiple image sets are arranged in time sequence; Calculating a timestamp difference between a target frame and adjacent frames in the plurality of image sets; wherein the target frame and the adjacent frames are adjacent in time sequence and belong to different image sets; When the timestamp differences are all within a preset threshold range, acquiring a preset number of adjacent images from the plurality of image sets; The preset number of adjacent images are input into a preset visual algorithm model to obtain the position of the personnel in the cabin output by the preset visual algorithm model. The preset visual algorithm model is a pre-trained model for identifying the position of personnel in the image.

2. The method according to claim 1, characterized in that The preset threshold range is a time range between a first preset threshold and a second preset threshold, and the first preset threshold is smaller than the second preset threshold; After calculating the timestamp difference between the target frame and adjacent frames in the plurality of image sets, the method further includes: When the timestamp difference is less than the first preset threshold, deleting the target frame from the first image set corresponding to the target frame to update the first image set; When the timestamp difference is greater than the second preset threshold, the adjacent frames are deleted from the second image set corresponding to the adjacent frames to update the second image set.

3. The method according to claim 2, characterized in that After deleting the adjacent frame from the second image set corresponding to the adjacent frame to update the second image set when the timestamp difference is greater than the second preset threshold, and before acquiring a preset number of adjacent frames from the multiple image sets when the timestamp differences are all within the preset threshold, the method further includes: For the updated first image set and the updated second image set, calculating an update timestamp difference between a pending frame and a pending adjacent frame until the update timestamp difference is within the preset threshold range; wherein the pending frame is any frame in the updated first image set; Timestamp differences corresponding to other image sets in the multiple image sets are calculated, where the other image sets are image sets other than the first image set and the second image set in the multiple image sets.

4. The method according to claim 1, wherein After acquiring the plurality of image sets and before calculating the time stamp difference between the target frame and adjacent frames in the plurality of image sets, the method further includes: sorting all image frames included in different image sets to which the target frame and the adjacent frames belong according to timestamp sizes of the image frames; All sorted image frames are stored in a queue, wherein the target frame and the adjacent frames in the queue are adjacent in sequence.

5. The method according to claim 1, wherein The training process of the preset visual algorithm model includes: Obtaining sample image sets corresponding to different camera orientation information and / or camera type information, and personnel location tags corresponding to the sample image sets; Inputting the sample image set into an initial visual algorithm model to obtain a predicted personnel position output by the initial visual algorithm model; Calculating a loss value between the predicted person position and the person position label corresponding to the sample image set based on a preset loss function; When the loss value is greater than the loss threshold, the model parameters of the initial visual algorithm model are adjusted until the loss value is less than or equal to the loss threshold, thereby obtaining the converged preset visual algorithm model.

6. The method according to any one of claims 1 to 5, characterized in that The camera orientation information includes front row and rear row; the camera type information includes infrared and color; The multiple image sets include at least two of the image sets corresponding to the front-row infrared camera, the image sets corresponding to the front-row color camera, the image sets corresponding to the rear-row infrared camera, and the image sets corresponding to the rear-row color camera.

7. A device for detecting people in a vehicle cabin, characterized in that: include: an acquisition module, configured to acquire a plurality of image sets, wherein different image sets correspond to different camera orientation information and / or different camera type information; and image frames included in the plurality of image sets are arranged in a time sequence; a calculation module, configured to calculate a timestamp difference between a target frame and adjacent frames in the plurality of image sets; wherein the target frame and the adjacent frames are adjacent in time sequence and belong to different image sets; The acquisition module is further configured to acquire a preset number of adjacent images from the plurality of image sets when the timestamp differences are all within a preset threshold range; The recognition module is used to input the target image into a preset visual algorithm model to obtain the position of the personnel in the cabin output by the preset visual algorithm model. The preset visual algorithm model is a pre-trained model for identifying the position of personnel in the image.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for detecting a person in a vehicle cabin according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting a person in a vehicle cabin according to any one of claims 1 to 6 is implemented.

10. A vehicle, characterized in that: include: A plurality of cameras of different orientations and / or types, and the device for detecting people in a vehicle cabin as claimed in claim 7, or the electronic device as claimed in claim 8.

Images