Vehicle state detection method and device

Abstract

The application discloses a vehicle state detection method and device. The vehicle state detection method comprises the following steps: detecting the door state of a to-be-processed vehicle in a current vehicle image in a monitoring video; in response to the door state being an open state, determining the difference information of the current vehicle image compared with a first reference vehicle image, the first reference vehicle image being a vehicle image before the current vehicle image in the monitoring video; and determining whether the door of the to-be-processed vehicle is open based on the difference information. The application compares the difference information of the current vehicle image and the first reference vehicle image to exclude the false detection of the vehicle door opening detection, thereby improving the recognition accuracy of the door state of the vehicle state.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a vehicle status detection method and device. Background Art

[0002] In daily traffic management, it is often necessary to detect the status of vehicle doors. However, the accuracy of most current door detection methods is not high, and errors often occur in the door status detection results. Summary of the Invention

[0003] The present application provides a vehicle status detection method and device, which can improve the accuracy of vehicle door detection.

[0004] To achieve the above objectives, the present application provides a vehicle status detection method, which includes:

[0005] Detect the door status of the vehicle to be processed in the current vehicle image in the surveillance video;

[0006] In response to the door being in the open state, determining difference information between the current vehicle image and a first reference vehicle image, where the first reference vehicle image is a vehicle image preceding the current vehicle image in the surveillance video;

[0007] Based on the difference information, it is determined whether the door of the vehicle to be processed is open.

[0008] Determining whether the door of the vehicle to be processed is open based on the difference information includes:

[0009] In response to the difference information being greater than a threshold, it is determined that a door of the vehicle to be processed is open.

[0010] Before determining whether the door of the vehicle to be processed is open based on the difference information, the following steps are further included:

[0011] It is determined that the door state of the vehicle to be processed in the first reference vehicle image is an open state.

[0012] After determining that the door of the vehicle to be processed is in an open state, the method further includes:

[0013] Perform pedestrian detection on the current vehicle image;

[0014] In response to detecting a pedestrian in the current vehicle image, determining that a target behavior exists in the surveillance video; the target behavior includes at least one of the following:

[0015] The act of pedestrians entering a vehicle to be processed;

[0016] The act of a pedestrian leaving a vehicle to be processed.

[0017] Among them, after performing pedestrian detection on the current vehicle image, it also includes:

[0018] In response to no pedestrian being detected in the current vehicle image, pedestrian detection is performed on a second reference vehicle image; the second reference vehicle image is a vehicle image subsequent to the current vehicle image in the surveillance video, and a door state of the vehicle to be processed in the second reference vehicle image is open;

[0019] In response to detecting a pedestrian in the second reference vehicle image, it is determined that a target behavior exists in the surveillance video.

[0020] After determining that the target behavior exists in the surveillance video, the following steps are also included:

[0021] Determine whether the vehicle to be processed is in a no-parking zone associated with the target scene;

[0022] In response to the vehicle to be processed being in a no-parking zone, it is determined that the target behavior is an illegal driving behavior associated with the target scenario.

[0023] After determining that a pedestrian has illegally entered or left a vehicle to be processed, the following procedures are also included:

[0024] Based on at least one of the license plate information, location information and driver information of the vehicle to be processed, an alarm message for illegal driving behavior is issued.

[0025] Among them, pedestrian detection is performed on the current vehicle image, including:

[0026] Expand the vehicle area in the current vehicle image to obtain a human body detection range. The human body detection area includes the vehicle area, and the size of the human body detection area is smaller than the current vehicle image.

[0027] In the human body detection range, pedestrian detection is performed.

[0028] Determining difference information between the current vehicle image and the first reference vehicle image includes:

[0029] Calculating texture feature values ​​of the current vehicle image and the first reference vehicle image;

[0030] Difference information is determined based on a difference in texture feature values ​​between the current vehicle image and the first reference vehicle image.

[0031] Detecting the door status of the vehicle to be processed in the current vehicle image in the surveillance video includes:

[0032] Input the current vehicle image into the trained door state detection model;

[0033] Using the feature extraction layer in the door state detection model, extract features from the current vehicle image to obtain a first feature map of the current vehicle image; and

[0034] The first feature map is convolved using the dilated convolution kernel in the dilated convolution layer in the door state detection model to obtain the door state of the vehicle to be processed; wherein the dilated convolution kernel includes a convolution kernel with a dilated factor added thereto.

[0035] Using the feature extraction layer in the door state detection model, feature extraction is performed on the current vehicle image to obtain a first feature map of the current vehicle image, including:

[0036] Using the feature extraction module in the feature extraction layer, extract features of the current vehicle image to obtain a second feature map of the current vehicle image;

[0037] The feature attention module in the feature extraction layer is used to determine the weights of each channel in the second feature map, and the second feature map is channel-weighted based on the weights of each channel to obtain the first feature map of the current vehicle image.

[0038] Before detecting the door status of the vehicle to be processed in the current vehicle image in the surveillance video, the following steps are included:

[0039] The vehicle images in the training set are detected using the door state detection model to obtain the predicted coordinate values ​​of the vehicle door key points in the vehicle images;

[0040] Calculate the key point prediction loss based on the predicted coordinate values ​​of the door key points;

[0041] The door state detection model is trained based on key point prediction loss.

[0042] The method further includes:

[0043] Calculate the class loss, confidence loss, and bounding box loss of the door state detection model;

[0044] The steps for training the door state detection model based on key point prediction loss include:

[0045] Calculate the total loss based on keypoint prediction loss, class loss, confidence loss and bounding box loss;

[0046] The door state detection model is trained based on the total loss.

[0047] To achieve the above objectives, the present application also provides an electronic device, which includes a processor; the processor is used to execute instructions to implement the above method.

[0048] To achieve the above objectives, the present application also provides a computer-readable storage medium for storing instructions / program data, which can be executed to implement the above method.

[0049] In the vehicle state detection method, the door state of a to-be-processed vehicle in a current vehicle image in a monitoring video is detected, and in a case where the door state detection result of the to-be-processed vehicle is an open state, difference information of the current vehicle image compared with a first reference vehicle image is confirmed, the first reference vehicle image being a vehicle image before the current vehicle image in the monitoring video, so as to verify the detection result of the door state of the to-be-processed vehicle based on the difference information of the current vehicle image compared with the first reference vehicle image, thereby excluding the false detection of the vehicle door opening detection through the comparison of the difference information of the current vehicle image and the first reference vehicle image, and improving the recognition accuracy of the door state of the vehicle state. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0051] Figure 1 FIG. 1 is a flowchart of an embodiment of the vehicle state detection method of the present application;

[0052] Figure 2 FIG. 3 is a schematic diagram of a dilated convolution layer of a door state detection model in the vehicle state detection method of the present application;

[0053] Figure 3 FIG. 4 is a structural schematic diagram of a door state detection model in the vehicle state detection method of the present application;

[0054] Figure 4 FIG. 6 is a schematic diagram of a texture feature map of a current vehicle image in the vehicle state detection method of the present application;

[0055] Figure 5 FIG. 7 is a schematic diagram of a texture feature map of a first reference vehicle image in the vehicle state detection method of the present application;

[0056] Figure 6 FIG. 8 is a schematic diagram of pedestrian detection in the vehicle state detection method of the present application;

[0057] Figure 7 FIG. 9 is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0058] Figure 8 FIG. 10 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application. In addition, unless otherwise specified (for example, "or in addition" or "or in an alternative"), the term "or" as used herein refers to a non-exclusive "or" (that is, "and / or"). Furthermore, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments.

[0060] The present application provides a vehicle status detection method, which determines the door status in a vehicle image through the comprehensive results of vehicle door status detection and vehicle image change detection. In this way, the detection results of the door status can be verified by using the difference information of the vehicle image to improve the recognition accuracy of the vehicle door status.

[0061] Specific as Figure 1 As shown, the vehicle status detection method of this embodiment includes the following steps. It should be noted that the following step numbers are only used to simplify the description and are not intended to limit the execution order of the steps. The execution order of the steps of this embodiment can be changed arbitrarily without violating the technical concept of this application.

[0062] S101: Detecting the door status of a vehicle to be processed in a current vehicle image in a surveillance video.

[0063] The door status of the vehicle to be processed in the current vehicle image in the surveillance video can be detected, so that when the door status detection result of the vehicle to be processed is an open state, the difference information between the current vehicle image and the first reference vehicle image can be subsequently confirmed, where the first reference vehicle image is the vehicle image before the current vehicle image in the surveillance video, so as to verify the detection result of the door status of the vehicle to be processed based on the difference information between the current vehicle image and the first reference vehicle image, so as to improve the recognition accuracy of the door status of the vehicle.

[0064] The current vehicle image may be detected using a door state detection model to detect whether the door state of the vehicle to be processed in the current vehicle image is open.

[0065] Among them, the door state detection model may include a feature extraction layer and a hole convolution layer. Among them, the feature extraction layer is used to extract features from the current vehicle image to obtain the first feature map of the current vehicle image. The hole convolution layer is used to perform convolution processing on the first feature map through the hole convolution kernel to obtain the door state of the vehicle to be processed, so as to improve the accuracy of target positioning while expanding the receptive field through the hole convolution layer without reducing the resolution, making the output more accurate, thereby solving the problem of low accuracy and easy missed detection in the door state detection process, and avoiding the problem of reduced spatial resolution and information loss in the solution of expanding the receptive field by downsampling. Among them, Figure 2 As shown in FIG, the dilated convolution kernel includes a convolution kernel with a dilation factor added thereto. The value of the dilation factor can be 0.

[0066] Optionally, the feature extraction layer includes a feature extraction module and a feature attention module. The feature extraction module is used to extract features from the current vehicle image to obtain a second feature map of the current vehicle image; the feature attention module is used to determine the weights of each channel in the second feature map, and perform channel weighting on the second feature map based on the weights of each channel to obtain a first feature map of the current vehicle image. In this way, by adding the feature attention module to the feature extraction layer of the door state detection model, the door state detection model pays more attention to the area where the door is open, so as to improve the accuracy of the door state detection. The feature attention module in the feature extraction layer of the door state detection model can be an SE attention (SElayer) module, so that the weights of channels with high importance become higher and the weights of channels with low importance become lower, so that the door state detection model pays more attention to the area where the door is open.

[0067] In addition, optionally, the type of the door state detection model is not limited, for example, it can be a YOLO model.

[0068] Specifically, if Figure 3 As shown, the YOLOv5 model can be used to construct a door state detection model, which may include a feature extraction (Backbone) layer, an intermediate (Neck) unit, and an output (Head) unit.

[0069] The above-mentioned dilated convolution layer can be set in the output unit or the intermediate unit. Specifically, the dilated convolution layer can be set at the head of the output unit, that is, the output unit can include a dilated convolution layer and at least one convolution layer connected to the dilated convolution layer. In this way, the dilated convolution layer is added to the head of the output unit of the door state detection model. Figure 2 The hole convolution layer shown on the right can expand the receptive field without reducing the resolution, thereby improving the accuracy of target positioning.

[0070] Furthermore, the Backbone unit utilizes a cross-stage local network structure (CSPNet) to extract rich semantic features from vehicle images. It also incorporates an attention module (SElayer) to focus the network on areas where doors are open. Compared to other convolutional neural network architectures, CSPNet addresses the issue of gradient information duplication within the network by integrating gradient changes from beginning to end into the feature map, reducing computational effort while ensuring model accuracy. Neck utilizes a path aggregation network (PANet) to generate feature pyramids and pass features to the prediction layer. The Neck unit mainly consists of four parts: (1) FPN feature pyramid, whose main function is to fuse semantic features at high and low resolutions to improve the effect of target detection, especially the detection results of small targets, such as vehicles in the distance with small open doors. The use of FPN can significantly improve the effect; (2) bottom-up path expansion, which makes it easier for the bottom-level information to be transmitted to the top of the high-level layer, reducing the amount of calculation; (3) adaptive pooling feature layer, whose purpose is to perform feature fusion to make the extracted ROI features richer; (4) fully connected layer, which introduces a fully connected branch for foreground and background classification for the original segmentation branch, and obtains more accurate results by fusing the outputs of these two branches.

[0071] In addition, before step S101 , the vehicle door state detection model may be trained so that the vehicle door state detection model has the capability of detecting the vehicle door state.

[0072] Before training, the vehicle images can be uniformly resized to a preset size, and the resizing is proportional, so that the image content will not be distorted due to the resizing. The preset size is not limited, for example, it can be 224*224.

[0073] In addition, data enhancement operations such as random rotation, mirroring, translation, and noise addition can be performed on vehicle images. After data enhancement operations, the size of the data set can be expanded to a certain extent, reducing the possibility of model overfitting.

[0074] Optionally, during the training process, the door state detection model can be trained based on the positional supervision of the door key points. Specifically, when using the door state detection model to detect a vehicle image, the output unit of the door state detection model can output not only the confidence that the vehicle door in the vehicle image is open and the confidence that the door is closed, but also the predicted coordinate values ​​of the key points of the vehicle door in the vehicle image. The predicted coordinate values ​​of the key points output by the door state detection model can be used to calculate the key point prediction loss of the door state detection model, and the key point prediction loss can then be used to train the door state detection model. The aforementioned door key points can include the four corner points of the door.

[0075] Among them, the calculation formula of the key point prediction loss L2 can be shown as follows:

[0076]

[0077] Among them, L2 is the key point prediction loss, m is the total number of key points, y (i) is the real coordinate value of the key point of the car door, Predict coordinate values ​​for the door key points.

[0078] In addition, during the training process, the class loss and confidence loss of the door state detection model can also be calculated to train the door state detection model using the class loss and confidence loss. The sum of the class loss and confidence loss of the door state detection model can be calculated using the following formula:

[0079]

[0080] Where n is the total number of training images, y i is the true confidence that the door belongs to each category, x i is the prediction confidence of the door belonging to each category, and BCEWithLogitsLoss is the sum of the class loss and confidence loss of the door state detection model. It is understandable that in other embodiments, the class loss and confidence loss of the door state detection module can be calculated using different formulas.

[0081] Optionally, during the training process, the position of the door detection frame can be used to supervise the training of the door state detection model. Specifically, when using the door state detection model to detect vehicle images, the output unit of the door state detection model can output not only the confidence that the vehicle door in the vehicle image belongs to the open category and the confidence that the door belongs to the closed category, but also the predicted rectangular frame of the vehicle door in the vehicle image. The predicted rectangular frame of the door can be used to calculate the bounding box loss of the door state detection model, and then the bounding box loss can be used to train the door state detection model.

[0082] The calculation formula of the bounding box loss can be shown as follows:

[0083]

[0084] Among them, A is the marked rectangle, B is the predicted rectangle, and C is the minimum circumscribed rectangle of the image formed by the two frames, that is, C contains the area of ​​the minimum frame of the predicted rectangle and the marked rectangle. GIOU is the bounding box loss.

[0085] Exemplarily, in the training process of the vehicle door state detection model, the bounding box loss, the class loss, the confidence loss and the key point prediction loss of the vehicle door state detection model can be calculated; the bounding box loss, the class loss, the confidence loss and the key point prediction loss are weighted to obtain the total loss of the vehicle door state detection model; and the total loss is used to train the vehicle door state detection model. In addition, in the training process, the weights of the class loss, the confidence loss, the bounding box loss and the key point loss can be adjusted to make the model converge and improve the robustness of the model.

[0086] In addition, the monitoring video is composed of multiple frames of original images. In step S101, vehicle detection can be performed on each frame of original image in the monitoring video. If a vehicle is detected, a vehicle image can be extracted from each frame of original image. Then, the vehicle door state detection model is used to detect the vehicle image to determine whether the vehicle door of the vehicle in the vehicle image is open. Specifically, the vehicle region in the original image can be extracted as the vehicle image. Alternatively, considering the factor of the open vehicle door, the vehicle region determined by detection can be expanded by a first percentage, and the vehicle image can be obtained by cropping. The first percentage can be set according to actual conditions, which is not limited herein, for example, 20%, 21%, 24%, 25%, 27%, or 30%.

[0087] Alternatively, if a vehicle is detected from the original image, it can be determined whether each vehicle detected from the original image is in a stopped state. Then, the vehicle image in the stopped state is extracted for vehicle door state detection to determine whether the vehicle door of the vehicle in the stopped state is open. The vehicle to be processed in the present application can be the vehicle in the stopped state.

[0088] Further, the vehicle image in the stopped state can be subjected to pedestrian detection to determine whether the vehicle in the stopped state has pick-up and drop-off behavior.

[0089] The method for determining whether the vehicle is stopped can be various, which is not limited herein. For example, whether the vehicle is stopped can be determined by determining whether the position of the vehicle changes.

[0090] S102: In response to the vehicle door state being open, the difference information of the current vehicle image compared with the first reference vehicle image is determined.

[0091] S103: Based on the difference information, it is determined whether the vehicle door of the vehicle to be processed is open.

[0092] Based on step S101 detecting that the door state of the vehicle to be processed in the current vehicle image is open, the difference information between the current vehicle image and the first reference vehicle image can be determined, wherein the first reference vehicle image is the vehicle image before the current vehicle image in the monitoring video, so as to verify the detection result of the door state of the vehicle to be processed based on the difference information between the current vehicle image and the first reference vehicle image, so as to eliminate the false detection of the vehicle door opening detection by comparing the difference information between the current vehicle image and the first reference vehicle image, so as to improve the recognition accuracy of the door state of the vehicle state.

[0093] The first reference vehicle image for the current vehicle image can be the vehicle image n frames before the current vehicle image. Specifically, it can be understood as the vehicle image at the corresponding position in the original image n frames before the original image to which the current vehicle image belongs. n can be set based on actual conditions, for example, to 1, 2, or 5, as long as the vehicle in the current vehicle image and the first reference vehicle image is the same vehicle.

[0094] In one implementation, in response to a door state detection result of the vehicle to be processed in the current vehicle image being different from the door state of the vehicle to be processed in the vehicle image frame immediately preceding the current vehicle image, that is, in response to confirming a change in the door state of the vehicle to be processed based on the current vehicle image, step S102 may be executed to confirm whether the door state of the vehicle to be processed in the current vehicle image is indeed different from the door state of the vehicle to be processed in the first reference vehicle image. Specifically, if the difference information between the current vehicle image and the first reference vehicle image is confirmed to be greater than a threshold, the door state of the vehicle to be processed in the current vehicle image and the door state of the vehicle to be processed in the first reference vehicle image are confirmed to be indeed different, and the detection result may be confirmed to be correct, thereby confirming the door state of the vehicle to be processed. If the difference information between the current vehicle image and the first reference vehicle image is confirmed to be less than a threshold, the door state of the vehicle to be processed in the current vehicle image and the door state of the vehicle to be processed in the first reference vehicle image are confirmed to be the same, and the detection result may be confirmed to be problematic and require correction.

[0095] For example, if the doors of the vehicle to be processed in the previous vehicle image are not open, and the door state detection result of the vehicle to be processed in the current vehicle image is open, difference information between the current vehicle image and the first reference vehicle image can be determined. If the difference information between the current vehicle image and the first reference vehicle image is determined to be greater than a threshold, it can be confirmed that the doors of the vehicle to be processed in the current vehicle image are open. If the doors of the vehicle to be processed in the previous vehicle image are open, and the door state detection result of the vehicle to be processed in the current vehicle image is closed, difference information between the current vehicle image and the first reference vehicle image can be determined. If the difference information between the current vehicle image and the first reference vehicle image is determined to be greater than a threshold, it can be confirmed that the doors of the vehicle to be processed in the current vehicle image are not open.

[0096] In this case, it is necessary to ensure that the door status of the selected first reference vehicle image is different from the door status of the vehicle to be processed in the current vehicle image. To prevent the door status of vehicles imaged within a seconds before the current vehicle image from being mistakenly detected as different from the door status in the current vehicle image, the time difference between the selected first reference vehicle image and the current vehicle image can be greater than a seconds to ensure that the door status of the current vehicle image is correctly confirmed. The value a can be set based on actual circumstances, for example, 1s, 1.5s, or 2s. Specifically, the time difference between the first reference vehicle image and the current vehicle image can be between 1s and 2s.

[0097] In addition, in response to confirming that the door state of the vehicle to be processed has not changed based on the door state detection result of the current vehicle image, that is, in response to the door state detection result of the vehicle to be processed in the current vehicle image being the same as the door state of the vehicle to be processed in the vehicle image frame before the current vehicle image, step S102 may be omitted, and the door state detection result of the vehicle to be processed in the current vehicle image may be directly used as the final door state of the vehicle to be processed in the current vehicle image. For example, if the door state of the vehicle to be processed in the vehicle image frame before the current vehicle image is closed, and the door state detection result of the vehicle to be processed in the current vehicle image is closed, the door state of the vehicle to be processed in the current vehicle image may be directly confirmed as closed.

[0098] In another implementation, in response to the door state detection result of the vehicle to be processed in the current vehicle image being different from the door state of the vehicle to be processed in the vehicle image of the previous frame of the current vehicle image, that is, in response to confirming that the door state of the vehicle to be processed has changed based on the current vehicle image, step S102 can be executed; in step S103, when it is confirmed that the difference information between the current vehicle image and the first reference vehicle image is greater than a threshold, it is confirmed that the door state of the vehicle to be processed in the current vehicle image is indeed different from the door state of the vehicle to be processed in the first reference vehicle image, then the detection result can be confirmed to be correct, and thus the door state of the vehicle to be processed can be confirmed; in step S103, when it is confirmed that the difference information between the current vehicle image and the first reference vehicle image is less than a threshold, it is confirmed that the door state of the vehicle to be processed in the current vehicle image is the same as the door state of the vehicle to be processed in the first reference vehicle image, then it can be confirmed that there is a problem with the detection result and the detection result needs to be corrected.

[0099] In response to the door status detection result of the vehicle to be processed in the current vehicle image being the same as the door status of the vehicle to be processed in the vehicle image of the previous frame of the current vehicle image, that is, in response to confirming that the door status of the vehicle to be processed has not changed based on the current vehicle image, step S102 can be executed; in step S103, when it is confirmed that the difference information between the current vehicle image and the first reference vehicle image is less than the threshold, it is confirmed that the door status of the vehicle to be processed in the current vehicle image is indeed the same as the door status of the vehicle to be processed in the first reference vehicle image, then the detection result can be confirmed to be correct, and thus the door status of the vehicle to be processed can be confirmed; when it is confirmed that the difference information between the current vehicle image and the first reference vehicle image is greater than the threshold, it is confirmed that the door status of the vehicle to be processed in the current vehicle image is different from the door status of the vehicle to be processed in the first reference vehicle image, then it can be confirmed that there is a problem with the detection result and the detection result needs to be corrected.

[0100] The difference information between the current vehicle image and the first reference vehicle image may be calculated in a variety of ways, such as the two calculation methods shown below.

[0101] First, the current vehicle image and the first reference vehicle image can be directly subtracted to obtain a difference image of the current vehicle image, and then a determination can be made based on the difference image of the current vehicle image whether the difference information is greater than a threshold. Specifically, the determination can be made based on the average pixel value of the difference image of the current vehicle image whether the average pixel value of the difference image is greater than the threshold. Preferably, to improve the accuracy of determining whether the difference information is greater than the threshold, the average pixel value of all pixels within the predicted rectangular box of the vehicle door in the difference pixels of the current vehicle image can be calculated, and then a determination can be made whether the average pixel value of all pixels within the predicted rectangular box is greater than the threshold. This allows for accurate determination of changes in the vehicle door area and improves the accuracy of door opening.

[0102] Secondly, the texture feature values ​​of the current vehicle image and the first reference vehicle image can be calculated respectively to obtain the following: Figure 4 The texture feature map of the current vehicle image shown and Figure 5 The texture feature map of the first reference vehicle image shown is then subtracted from the texture feature map of the current vehicle image to obtain a texture feature difference image. Subsequently, based on the texture feature difference image of the current vehicle image, it is determined whether the difference information is greater than a threshold. Using change detection based on texture features, false detections of vehicle door opening are eliminated, further improving the accuracy of door opening detection and avoiding situations where the door image and the background image are not significantly different, resulting in the texture feature difference image being unable to represent changes in the door state, thereby leading to erroneous door state determination. Specifically, based on the average pixel value of the texture feature difference image of the current vehicle image, it is determined whether the average pixel value of the texture feature difference image is greater than a threshold. Preferably, to improve the accuracy of determining whether the difference information is greater than the threshold, the average pixel value of all pixels within the predicted rectangular box of the door in the texture feature difference pixels of the current vehicle image can be calculated, and then it is determined whether the average pixel value of all pixels within the predicted rectangular box in the texture feature difference image is greater than the threshold. This allows accurate determination of changes in the door area and improves the accuracy of door opening.

[0103] The above threshold value can be set according to actual conditions and is not limited here. For example, it can be 20.

[0104] Optionally, the above vehicle status detection method can be applied to the detection of pedestrians entering or leaving the vehicle (ie, pedestrian getting on and off the vehicle detection).

[0105] As an example, detecting a pedestrian entering or leaving a vehicle can be broken down into two parts: pedestrian detection and vehicle door opening / closing motion recognition. A pedestrian entering or leaving a vehicle can only be determined if the vehicle door opening / closing motion is recognized and a pedestrian is detected near the vehicle with the door open.

[0106] In existing technologies, action recognition is typically based on manually designed features. For example, features such as Scale Invariant Feature Transform (SIFT) descriptors, 3-dimensional Histogram of Oriented Gradient (HOG3D) descriptors, and Speed ​​Up Robust Features (SURF) descriptors are extracted between consecutive frames and fed into a classifier to identify specific actions in the dataset. However, these methods of action recognition based on manually designed features have low accuracy and are time-consuming, making them difficult to adopt in practice.

[0107] Based on this, the present application determines whether a pedestrian enters or leaves the vehicle through the comprehensive results of vehicle status and pedestrian detection, which greatly reduces the workload of supervisors. There is no need to detect pedestrians entering or leaving the vehicle based on specific features involved in manual work, and the difference information of the vehicle image is used to verify the detection results of the door status, avoiding the situation where the detection results of the door status are incorrect, resulting in errors in the detection of pedestrians entering or leaving the vehicle, thereby improving the high-precision recognition of the behavior of pedestrians entering or leaving the vehicle.

[0108] Optionally, based on the above solution, when it is confirmed that the door is in an open state in the current vehicle image, as shown in FIG. Figure 6 As shown, pedestrian detection can be performed on the current vehicle image to determine whether a pedestrian has engaged in a target behavior based on the current vehicle image. The target behavior includes at least one of the following: a pedestrian entering the vehicle to be processed; a pedestrian leaving the vehicle to be processed. Specifically, if a pedestrian is detected in the current vehicle image, it can be determined that the pedestrian has entered or left the vehicle to be processed. If no pedestrian is detected in the current vehicle image, pedestrian detection can be continued on a second reference vehicle image to determine whether a pedestrian has entered or left the vehicle to be processed based on a second reference vehicle image subsequent to the current vehicle image.

[0109] Among them, the above-mentioned second reference vehicle image refers to the k frames of vehicle images after the current vehicle image, and the current vehicle image and the vehicle in each frame of the second reference vehicle image are the same vehicle, and the doors of the vehicles in the current vehicle image and each frame of the second reference vehicle image are in an open state. In this way, based on step S103, if it is detected that during the process of opening the door of the vehicle in the current vehicle image, a pedestrian appears in the vehicle range or the human body detection range defined by the vehicle image, it can be explained that there is a passenger getting on and off the vehicle in the current vehicle image, that is, the pedestrian has entered or left the vehicle to be processed.

[0110] The above k is determined based on the open / closed state of the vehicle's doors. If the first vehicle image after the current vehicle image shows a closed vehicle door, which is the sixth vehicle image after the current vehicle image, then k is 5, meaning that the five vehicle images after the current vehicle image are all second reference vehicle images. k can also be 0, for example, if the vehicle's doors were only opened for a moment and were already closed in the first image after the current vehicle image.

[0111] Among them, the above-mentioned step of "performing pedestrian detection on vehicle images" may include: determining a human body detection range based on the vehicle image; and performing pedestrian detection on the determined human body detection range to improve the detection accuracy of pedestrians entering or leaving the vehicle.

[0112] In one implementation, the vehicle image may be directly used as the human body detection range, that is, step S103 may directly perform pedestrian detection on the vehicle image.

[0113] In another implementation, the vehicle area in the vehicle image can be determined, and then the vehicle area can be expanded to obtain a human detection area. The human detection area can include the vehicle area. And the size of the human detection area can be smaller than the size of the vehicle image, or smaller than the size of the original image to which the vehicle image belongs. In a specific example, the vehicle area can be expanded by expanding the vehicle area by a preset percentage, and then the expanded vehicle area is used as the human detection range. The preset percentage can be set according to actual conditions and is not limited here. For example, it can be 20%-30%, specifically 25%. In another specific example, the vehicle area can be expanded by pulling the border of the vehicle area outward.

[0114] In another embodiment, the vehicle image may be directly expanded to obtain the human detection area. In this solution, the vehicle image may be expanded by expanding by a second percentage or by pulling the border of the vehicle image outward.

[0115] In addition, the expansion mentioned in this application may refer to extracting an expanded image from the original image to which the vehicle image belongs.

[0116] Optionally, pedestrian detection may be performed on the vehicle image using a human body detection model to determine whether a pedestrian has entered or left the vehicle at the moment the vehicle image belongs.

[0117] In this embodiment, the comprehensive results of vehicle door status detection, vehicle image change detection and pedestrian detection are used to determine whether pedestrians are getting on or off the vehicle, which greatly reduces the workload of supervisors. There is no need to perform pedestrian getting on or off detection based on specific features involved in manual work, and the difference information of the vehicle image is used to verify the detection results of the door status, avoiding the situation where the detection results of the door status are incorrect, resulting in incorrect getting on or off behavior detection, thereby improving the high-precision recognition of pedestrian getting on or off the vehicle.

[0118] In addition, if in step S101, it is detected that the door in the current vehicle image is in a closed state, but the door of the vehicle in the previous frame vehicle image of the current vehicle image is in an open state, the difference information of the current vehicle image and the first reference vehicle image can also be compared to determine whether the difference information is greater than a threshold. If it is greater than the threshold, it is determined that the door state of the vehicle in the current vehicle image has changed, so that it can be determined that the door in the current vehicle image is in a closed state, which can avoid pedestrians blocking the door and causing the open door to be detected as belonging to the closed category, that is, prevent the door state detection result from being erroneous, resulting in erroneous getting on and off the vehicle detection result, so as to improve the accuracy of getting on and off the vehicle detection.

[0119] Furthermore, the above-mentioned detection scheme for pedestrians entering or leaving a vehicle can be applied to the detection of illegal driving behaviors. Specifically, when a pedestrian is determined to have entered or left a vehicle in a prohibited parking zone associated with a target scenario, it is determined that the vehicle to be processed has engaged in illegal driving behaviors associated with the target scenario. When it is determined that illegal driving behaviors associated with the target scenario have occurred, the license plate and vehicle model of the vehicle corresponding to the illegal driving behaviors can be recognized, and at least one of the license plate information, vehicle information (such as vehicle model or vehicle location), and driver information of the vehicle corresponding to the illegal driving behaviors can be saved. Furthermore, based on at least one of the license plate information, vehicle information (such as vehicle model or vehicle location), and driver information of the vehicle that engaged in the illegal driving behaviors, an alarm message regarding the illegal driving behaviors can be issued.

[0120] Among them, the target scene mentioned in this application can be a school scene, a hospital scene, a scenic spot scene, a road scene or other scenes.

[0121] The no-parking zones for a target scene are areas where parking is prohibited. These zones may vary for different target scenes. For example, in a road scene, these zones may be traffic lights or lanes. For a school scene, these zones may include basketball courts, track and field venues, dormitory entrances, and other areas.

[0122] The illegal driving behaviors associated with the target scene may include: behaviors of pedestrians entering or leaving a vehicle parked in a no-parking zone in the target scene.

[0123] In one case, if the camera range of the camera device is a no-parking zone, then when the vehicle status detection method is used to determine that pedestrians are getting on or off the vehicle in the surveillance video taken by the camera device (i.e., pedestrians entering or leaving the vehicle), then the vehicle corresponding to the pedestrian getting on or off the vehicle has committed illegal driving behavior.

[0124] In another case, if a portion of the camera range of the camera device is a no-parking area, but the remaining portion is not a no-parking area, the image of the vehicle in the no-parking area can be extracted in step S101 to perform door status detection and pedestrian detection to determine whether the vehicle in the no-parking area has any illegal driving behavior.

[0125] The following will use a road scenario as an example to provide a detailed description of how to confirm whether a vehicle's parking location is a prohibited parking area.

[0126] Among them, it can be confirmed whether the parking location of the vehicle is a prohibited parking area based on preset traffic rules and the parking location of the vehicle.

[0127] More specifically, the method for determining a no-parking zone may include: if a vehicle is parked on a sidewalk or a non-motorized vehicle lane, and the parking position is not within the parking space frame, and there is no parking sign indicating that parking is allowed at the parking position, then the parking position is confirmed to be a no-parking zone; if the vehicle is parked on a motor vehicle lane, and the signal light indicating the vehicle's movement is green, and the number of vehicles on the motor vehicle lane is less than a second threshold, then the parking position is confirmed to be a no-parking zone; of course, no-parking zones other than the above-mentioned no-parking zones may also be determined through other implementation methods.

[0128] In addition, if the distance between the vehicle's parking position and the lane is greater than a first threshold, the vehicle's parking position is confirmed to be not a no-parking zone; or, if the vehicle is parked on a sidewalk or non-motorized vehicle lane and the vehicle's parking position is within the parking space line frame, the vehicle's parking position is confirmed to be not a no-parking zone; or, if the vehicle is parked on a sidewalk or non-motorized vehicle lane, the vehicle is not parked within the parking space line frame, but there is a parking sign indicating that parking is permitted at the vehicle's parking position, the parking position is confirmed to be not a no-parking zone; or, if the vehicle is parked in a motor vehicle lane, the traffic light indicating vehicle movement is red, and the distance between the vehicle and the vehicle in front of it or the zebra crossing is less than a third threshold, the parking position is confirmed to be not a no-parking zone; or, if the vehicle is parked in a motor vehicle lane, the traffic light indicating vehicle movement is green, and the number of vehicles in the motor vehicle lane is greater than a second threshold, that is, the motor vehicle lane is congested, the parking position is confirmed to be not a no-parking zone.

[0129] The first threshold, the second threshold and the third threshold can be set according to actual conditions and are not limited here.

[0130] See also Figure 7 , Figure 7 2 is a schematic diagram of the structure of an embodiment of the electronic device 20 of the present application. The electronic device 20 of the present application includes a processor 22, which is used to execute instructions to implement the method of any of the above embodiments of the present application and any non-conflicting combination thereof.

[0131] The electronic device 20 may be a camera device, a server or other device, which is not limited here.

[0132] The processor 22 may also be referred to as a CPU (Central Processing Unit). The processor 22 may be an integrated circuit chip having signal processing capabilities. The processor 22 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor 22 may be any conventional processor.

[0133] The electronic device 20 may further include a memory 21 for storing instructions and data required for the processor 22 to operate.

[0134] See also Figure 8 , Figure 8 Schematic diagram of the structure of the computer-readable storage medium in the embodiment of the present application. The computer-readable storage medium 30 of the embodiment of the present application stores instruction / program data 31, which, when executed, implements the method provided by any embodiment of the above-mentioned method of the present application and any non-conflicting combination. Among them, the instruction / program data 31 can form a program file and be stored in the above-mentioned storage medium 30 in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of the various embodiments of the present application. The aforementioned storage medium 30 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer, server, mobile phone, tablet and other devices.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0136] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0137] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0138] The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A vehicle status detection method, characterized in that: The vehicle state detection method comprises: Detect the door status of the vehicle to be processed in the current vehicle image in the surveillance video; In response to the door being in an open state, determining difference information between the current vehicle image and a first reference vehicle image, where the first reference vehicle image is a vehicle image preceding the current vehicle image in the surveillance video; Based on the difference information, determining whether a door of the vehicle to be processed is open; In response to a door of the vehicle to be processed being opened, performing pedestrian detection on the current vehicle image; In response to detecting a pedestrian in the current vehicle image, it is determined that a target behavior exists in the surveillance video; the target behavior includes at least one of the following: the behavior of the pedestrian entering the vehicle to be processed; the behavior of the pedestrian leaving the vehicle to be processed.

2. The vehicle state detection method according to claim 1, characterized in that: The determining, based on the difference information, whether the door of the vehicle to be processed is open includes: In response to the difference information being greater than a threshold, it is determined that a door of the vehicle to be processed is open.

3. The vehicle state detection method according to claim 1, characterized in that: Before determining whether the door of the vehicle to be processed is open based on the difference information, the method further includes: Determine that the door state of the vehicle to be processed in the first reference vehicle image is the open state.

4. The vehicle state detection method according to claim 1, characterized in that: After performing pedestrian detection on the current vehicle image, the method further includes: In response to not detecting the pedestrian in the current vehicle image, performing pedestrian detection on a second reference vehicle image; the second reference vehicle image is a vehicle image subsequent to the current vehicle image in the surveillance video, and the door state of the vehicle to be processed in the second reference vehicle image is the open state; In response to detecting a pedestrian in the second reference vehicle image, it is determined that the target behavior exists in the surveillance video.

5. The vehicle state detection method according to claim 1 or 4, characterized in that: After determining that the target behavior exists in the surveillance video, the method further includes: Determining whether the vehicle to be processed is in a no-parking zone associated with the target scene; In response to the vehicle to be processed being in the no-parking zone, it is determined that the target behavior is an illegal driving behavior associated with the target scenario.

6. The vehicle state detection method according to claim 5, characterized in that: After determining that the target behavior exists in the surveillance video, the method further includes: Based on at least one of the license plate information, location information and driver information of the vehicle to be processed, an alarm message for the illegal driving behavior is issued.

7. The vehicle state detection method according to claim 1, characterized in that: The performing pedestrian detection on the current vehicle image includes: Expanding the vehicle region in the current vehicle image to obtain a human body detection region; the human body detection region includes the vehicle region, and the size of the human body detection region is smaller than the current vehicle image; In the human body detection area, pedestrian detection is performed.

8. The vehicle state detection method according to claim 1, characterized in that: The determining difference information between the current vehicle image and the first reference vehicle image includes: Calculating texture feature values ​​of the current vehicle image and the first reference vehicle image; The difference information is determined based on a difference in texture feature values ​​between the current vehicle image and the first reference vehicle image.

9. The vehicle state detection method according to claim 1, characterized in that: The detecting of the door status of the vehicle to be processed in the current vehicle image in the monitoring video includes: Inputting the current vehicle image into the trained door state detection model; Using the feature extraction layer in the door state detection model, extract features from the current vehicle image to obtain a first feature map of the current vehicle image; and The first feature map is convolved using a dilated convolution kernel in a dilated convolution layer in the door state detection model to obtain the door state of the vehicle to be processed; wherein the dilated convolution kernel includes a convolution kernel with an added dilated factor.

10. The vehicle state detection method according to claim 9, characterized in that: The step of extracting features from the current vehicle image using the feature extraction layer in the door state detection model to obtain a first feature map of the current vehicle image includes: Using a feature extraction module in the feature extraction layer, extract features from the current vehicle image to obtain a second feature map of the current vehicle image; The feature attention module in the feature extraction layer is used to determine the weights of the channels in the second feature map, and the second feature map is channel-weighted based on the weights of the channels to obtain the first feature map of the current vehicle image.

11. The vehicle state detection method according to claim 10, characterized in that: Before detecting the door status of the vehicle to be processed in the current vehicle image in the surveillance video, the method includes: Using the door state detection model to detect the vehicle images in the training set, and obtain the predicted coordinate values ​​of the vehicle door key points in the vehicle images; Calculating the key point prediction loss based on the predicted coordinate values ​​of the door key points; The door state detection model is trained based on the key point prediction loss.

12. The vehicle state detection method according to claim 11, characterized in that: The method further comprises: Calculating the class loss, confidence loss, and bounding box loss of the door state detection model; The step of training the door state detection model based on the key point prediction loss includes: Calculating a total loss based on the keypoint prediction loss, the class loss, the confidence loss, and the bounding box loss; The door state detection model is trained based on the total loss.

13. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is configured to execute instructions to implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a program and / or instruction stored thereon, characterized in that: When the program and / or instructions are executed, the steps of the method according to any one of claims 1 to 12 are implemented.

Images