Vehicle window drop detection method and vehicle intelligent security inspection system and device

Abstract

The present invention discloses a vehicle window drop detection method and a vehicle intelligent security inspection system and device, which belong to the field of mixed security inspection of people and vehicles and are applied to a vehicle intelligent security inspection system. The vehicle intelligent security inspection system comprises at least four cameras, and the vehicle intelligent security inspection system is used to perform security inspection on the vehicle. The vehicle comprises four windows, and each of the windows is photographed by one of the cameras. The vehicle window drop detection method comprises: when a license plate on the vehicle is detected in a security inspection area, an image of the side of the vehicle is acquired; based on the image of the side of the vehicle, a window recognition model is established, and the state of the window is determined according to the image of the window and the window recognition model. By detecting the state of the vehicle window when the license plate on the vehicle is detected, manual detection of the vehicle window can be avoided, thereby improving the efficiency of security inspection, thereby shortening the security inspection process and saving police resources.

Description

Technical Field

[0001] The present invention belongs to the field of mixed security inspection of people and vehicles, and specifically relates to a vehicle window drop detection method and a vehicle intelligent security inspection system and device. Background Art

[0002] With the continuous advancement of science and technology, smart public security checkpoints, namely "smart security checks", are increasingly widely used in the field of public safety. They have realized rapid verification of people and vehicles based on the contactless collection of intelligent front-end equipment and the comparison capabilities of big data platforms, greatly improving the efficiency of security checks. The inspected vehicles can complete security checks in 8-10 seconds, realizing mixed security checks of people and vehicles in a "no card swiping, short stay" mode.

[0003] However, in the actual application of smart security inspection, there are still some practical problems. Restricted by existing video and perception technologies, all windows of the inspected vehicle must be lowered when arriving at the inspection area to ensure the collection of driver and passenger information in order to complete the subsequent personnel verification process. Whether the windows are lowered and whether they meet the collection requirements mainly depends on manual inspection by on-duty police officers. The manual inspection of windows causes the security inspection process to be time-consuming, and at the same time brings great adjustments to the already scarce police resources, affecting the overall application effect of smart security inspection. Summary of the invention

[0004] The purpose of the present invention is to provide a vehicle window drop detection method and a vehicle intelligent security inspection system and device, which can solve the problem that the manual inspection of vehicle windows prolongs the time-consuming security inspection process, and at the same time brings great adjustments to the already scarce police resources, affecting the overall application effect of intelligent security inspection.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present application provides a vehicle window drop detection method, which is applied to a vehicle intelligent security inspection system, wherein the vehicle intelligent security inspection system includes at least four cameras, and the vehicle intelligent security inspection system is used to perform security inspection on the vehicle, wherein the vehicle includes four windows, each of which is photographed by one of the cameras, and the vehicle window drop detection method includes:

[0007] When a license plate on a vehicle is detected in the security inspection area, an image of the side of the vehicle is acquired;

[0008] Based on the image of the side of the vehicle, a vehicle window recognition model is established, and the state of the vehicle window is determined according to the image of the vehicle window and the vehicle window recognition model, wherein the state of the vehicle window includes: the vehicle window is in an open state and the vehicle window is in a closed state.

[0009] Optionally, acquiring the image of the side of the vehicle includes:

[0010] Capturing a target video when the vehicle passes through a security inspection area, the target video includes multiple side images of the vehicle, the side images of the vehicle include multiple frames of images of vehicle windows of the vehicle, and selecting at least one frame including a side image of the target vehicle from the target video, and each side image of the target vehicle includes a vehicle window portion;

[0011] At a preset time interval, selecting a specific frame as a side image of the vehicle;

[0012] Processing the selected vehicle side image, and locating the vehicle window area from the vehicle side image using image segmentation technology;

[0013] Extracting the outline, area, and shape features of the vehicle window from the located vehicle window area;

[0014] The acquiring an image of a window of the vehicle based on the image of the side body, and determining a state of the window according to the image of the window, comprises:

[0015] Locating the image of the vehicle window in the side image, determining the state of the vehicle window according to the image of the vehicle window, and if the vehicle window is in an open state, stopping selecting the side image of the vehicle from the target video, and if the vehicle window is in a closed state, continuing to select the side image of the vehicle from the target video until it is determined that the vehicle window is in an open state;

[0016] The expression for selecting a specific frame as the side image of the vehicle is:

[0017] ;

[0018] in, Indicates Frame image, in a video stream, each frame is a snapshot at a point in time. Refers to the video frames selected at a certain time interval. Represents a single frame in a video stream, Indicates the frame number. Represents the time interval, which is the time difference between two consecutive frames and is used to define the frequency of selecting frames.

[0019] Optionally, when a license plate on a vehicle is detected, before acquiring an image of the side of the vehicle, the vehicle window drop detection method further includes:

[0020] Acquire multiple images of the side of a sample vehicle, wherein the images of the side of the sample vehicle include sample images of vehicle windows of the sample vehicle, and the sample images of the vehicle windows include images of the vehicle windows in an open state and images of the vehicle windows in a closed state;

[0021] Extracting features of the window glass from the sample image of the window of the sample vehicle, including the proportion of the window glass in the image, the shape of the window, and the clarity of the edge, so as to identify the outline of the window;

[0022] Based on the images of the side of the plurality of sample vehicles, a convolutional neural network is used to establish a vehicle window recognition model, wherein the vehicle window recognition model is used to recognize the state of the vehicle window, and during the training process, the marked sample images are used to learn the features of the open and closed states;

[0023] The determining the state of the vehicle window according to the image of the vehicle window comprises:

[0024] Recognize the image of the selected vehicle window according to the vehicle window recognition model, and determine the state of the vehicle window;

[0025] Perform anomaly detection, analyze the occlusion or damage of the window, and determine the corresponding response mechanism, including: analyzing the texture and color characteristics of the window area to determine whether there is an object blocking the window. If occlusion is detected, further determine whether an alarm or manual intervention is needed based on the degree of occlusion and the characteristics of the obstruction. Check whether the edge of the window area is intact and whether there are signs of cracks, breakage or damage. If damage is detected, record and report it, and further manual verification or processing.

[0026] Optionally, determining the state of the vehicle window according to the image of the vehicle window includes:

[0027] constructing virtual coordinates corresponding to the vehicle according to the selected image of the vehicle window, the virtual coordinates including coordinates of the vehicle body and coordinates of the vehicle window;

[0028] Determine a target ratio of the coordinates of the vehicle window to the coordinates of the vehicle body, and set a threshold to analyze the relationship between the target ratio and the threshold;

[0029] If the target ratio is greater than or equal to a preset threshold, it is determined that the window of the vehicle is in an open state; if the target ratio is less than the preset threshold, it is determined that the window of the vehicle is in a closed state.

[0030] Optionally, the process of detecting anomalies and analyzing the obstruction or damage of vehicle windows includes:

[0031] The collected images are grayed, filtered and denoised to improve the accuracy of subsequent processing. The texture features are extracted using the gray-level co-occurrence matrix, which is expressed as:

[0032] ;

[0033] in, Indicated in angle Next, grayscale and The frequency of occurrence, Represents the total number of pixel pairs, represents a function of pixel pairs, Indicates the position in the image The pixel gray value, To determine the location and Whether the pixel pair meets the gray level condition and ;

[0034] Extract color histogram in color space, extract color features, and analyze the color distribution of the image;

[0035] Compare the texture and color features of the current image with the baseline features of a normal car window and use a statistical test to detect occlusion, which is expressed as:

[0036] ;

[0037] in, Represents the chi-square statistic, which is used to compare actual observations and expected value The difference between represents the actual observed value, represents the expected value under normal conditions;

[0038] Calculate the ratio of the blocked area to the total window area to assess the degree of blockage;

[0039] The Canny edge detection algorithm is used to identify the integrity of the window edge. The expression is:

[0040] ;

[0041] in, Indicates at location The edge strength, and Respectively indicate the location The Sobel operator calculation results in the horizontal and vertical directions at are used for edge detection. Represents a small positive number, used to avoid division by zero during calculations;

[0042] Cracks and breakages are identified by analyzing the discontinuity and fragmentation of the edges, and the expression is:

[0043] ;

[0044] in, Represents the crack score, which is used to quantify the degree of cracks or fragmentation in the image. Indicates the total number of cracked or broken areas detected, Indicates The characteristics of the crack or broken area, Represents the cumulative characteristics of all cracked or broken areas;

[0045] Determine whether to issue an alarm or perform manual intervention based on the degree of obstruction and crack characteristics.

[0046] Optionally, after acquiring the image of the side of the vehicle, the vehicle window drop detection method further includes:

[0047] If a license plate other than the current vehicle is detected, the detection of the state of the window of the current vehicle is stopped.

[0048] Optionally, the vehicle window being in an open state comprises: a ratio of an area of ​​the vehicle window glass within a vehicle window frame to an area of ​​the vehicle window frame is less than or equal to one third.

[0049] Optionally, determining the state of the vehicle window according to the image of the vehicle window includes:

[0050] Applying a human figure detection algorithm to search for a human figure in the image, if a human figure image is detected in the image of the vehicle window, proceeding to the next step of analysis;

[0051] Extracting candidate human figure regions in the image of the vehicle window, and analyzing the shape descriptor of each candidate human figure region in combination with the boundary length and the region area;

[0052] The human figure recognition model is trained through the convolutional neural network model. The human figure recognition model is applied to each screened human figure candidate area to calculate the probability that it belongs to a human figure. The expression is:

[0053] ;

[0054] in, Represents a given feature vector When , the probability that the candidate region belongs to a human shape is, The feature vector representing the candidate region contains various features used for human recognition. Represents the activation function, which is used in the output layer of the classification model to convert the linear output into a probability value. Represents the weight vector of the model, indicating the contribution of the feature to the classification result. Represents the bias term, which is used to control the offset of the classification threshold;

[0055] According to the output probability of the human figure recognition model, combined with the preset threshold, it is determined whether a human figure image is detected, and then the window status is comprehensively inferred. The expression is:

[0056] ;

[0057] ;

[0058] in, Represents the comprehensive evaluation value, which is calculated by The single value calculated and summed is used to evaluate the state of the window and determine whether the window is open. Indicates the position in the image The strength of the humanoid features at Represents the binary representation of the window area. If the position In the window area, is equal to 1; otherwise, it is equal to 0, which is used to limit the calculation to the window area. Indicates location The shortest distance to the nearest window edge is used to evaluate whether the detected human figure is close to the window edge. Represents the threshold of the distance between the human figure and the edge of the car window. If Less than this threshold , it is believed that the figure is likely outside the car window, so the car window may be open. Represents a very small positive number to prevent the denominator from being zero. Represents an integer greater than 1, used to magnify the distance The impact of the results, An image representing the entire analysis, represents the coordinates of any position in the image, represents the set of all edge pixels, Indicates the coordinates of edge pixels, indicating the edge position of the window, Indicates in the collection Find the edge pixel that minimizes the distance formula. It should be noted that when the detected human feature has a high intensity and is located in the window area and close to the window edge, The value of will be relatively large because the humanoid feature strength and will contribute a larger value. On the contrary, if the human feature is weak or far away from the edge of the window, The value of will be smaller;

[0059] If it is inferred that a human figure is detected in the image of the vehicle window, the state of the vehicle window is marked as an open state.

[0060] Optionally, the vehicle window drop detection method further includes:

[0061] If the vehicle window is in an open state, a corresponding face image of the person inside the vehicle is collected;

[0062] If the vehicle window is in a closed state, a prompt message is issued, wherein the prompt message is used to instruct the vehicle window to switch from a closed state to an open state until the vehicle window is in an open state.

[0063] In a second aspect, an embodiment of the present application provides a vehicle intelligent security inspection system, which includes a controller and at least four cameras, wherein the cameras are electrically connected to the controller, and the controller is used to implement a vehicle window drop detection method.

[0064] In a third aspect, an embodiment of the present application provides a vehicle window drop detection device, the vehicle window drop detection device comprising:

[0065] A detection module, for acquiring an image of the side of the vehicle when a license plate on the vehicle is detected;

[0066] The determination module is used to obtain an image of a window of the vehicle based on the image of the side body, and to perform a vehicle window fall detection method according to the image of the window to determine the state of the window.

[0067] Beneficial effects of the present invention:

[0068] In the present application, when the license plate on the vehicle is detected, an image of the side of the vehicle is obtained; based on the image of the side, an image of the window is obtained, and the state of the window is determined according to the image of the window, wherein the state of the window includes: the window is in an open state and the window is in a closed state. That is, in the embodiment of the present application, when the license plate on the vehicle is detected, the state of the window of the vehicle is detected, thereby avoiding manual detection of the window of the vehicle, and improving the efficiency of security inspection, thereby shortening the security inspection process and saving police resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flow chart showing a method for detecting a fallen window of a vehicle provided in an embodiment of the present application for obtaining an image of the side of the vehicle;

[0070] Figure 2 A logic diagram showing a vehicle window drop detection method provided by an embodiment of the present application;

[0071] Figure 3 A schematic diagram showing a vehicle intelligent security inspection device provided in an embodiment of the present application;

[0072] Figure 4 A schematic diagram showing an electronic device a provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram showing the hardware structure of an electronic device b provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0075] The vehicle window drop detection method provided by the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0076] The embodiment of the present application provides a vehicle window drop detection method, which is applied to a vehicle intelligent security inspection system. The vehicle intelligent security inspection system includes at least four cameras. The vehicle intelligent security inspection system is used to perform security inspection on the vehicle. The vehicle includes four windows, each of which is photographed by a camera. The vehicle window drop detection method includes:

[0077] Step 101: When a license plate on a vehicle is detected in a security inspection area, an image of the side of the vehicle is acquired.

[0078] In actual application, the vehicle intelligent security inspection system includes a controller and five cameras, one of which is a license plate capture camera and the other four are wide-angle video cameras. Once the vehicle drives to the vehicle intelligent security inspection system, the license plate collection camera can collect the vehicle's license plate and send the collected license plate to the controller. Among them, the four window collection cameras are facing the four windows of the vehicle respectively. After receiving the license plate information of the vehicle, the controller can control the four license plate collection cameras to collect videos of the four windows of the vehicle respectively, and the videos include images of the side of the vehicle.

[0079] Step 102: Based on the image of the side of the vehicle, a vehicle window recognition model is established, and the state of the vehicle window is determined according to the image of the vehicle window and the vehicle window recognition model, wherein the state of the vehicle window includes: the vehicle window is in an open state and the vehicle window is in a closed state.

[0080] After receiving the license plate information of the vehicle, the controller can control the four license plate acquisition cameras to respectively capture videos of the four windows of the vehicle, wherein the videos include images of the side of the vehicle, so that the controller can locate the images of the vehicle windows from the images of the side of the vehicle, and thus the controller can determine the status of the windows through the images of the windows.

[0081] It should be noted that once the license plate on the vehicle is detected, it indicates that the vehicle has entered the detection range of the vehicle intelligent security inspection system. For example, when the vehicle enters the detection range of the vehicle intelligent security inspection system, the license plate capture camera can capture the image of the vehicle's license plate and send the captured license plate image to the controller, and the controller can detect the vehicle's license plate, thereby determining that the vehicle has entered the detection range of the vehicle intelligent security inspection system, and the controller can control the four cameras to respectively capture the four windows of the vehicle, and the controller can obtain the image of the window to determine the state of the window, and determine whether the window is open or closed.

[0082] In addition, in some implementations, the implementation method of obtaining the side image of the vehicle may be, referring to Figure 1 : Capturing a target video when a vehicle passes through a security inspection area, the target video includes multiple side images of the vehicle, the side images of the vehicle include multiple frames of images of vehicle windows, selecting at least one frame including a side image of the target vehicle from the target video, and each side image of the target vehicle includes a vehicle window portion;

[0083] At a preset time interval, a specific frame is selected as a side image of the vehicle;

[0084] Processing the selected vehicle side image, and locating the vehicle window area from the vehicle side image using image segmentation technology;

[0085] Extracting the outline, area, and shape features of the window from the located window area;

[0086] Based on the side image, an image of a vehicle window is acquired, and the state of the vehicle window is determined according to the image of the vehicle window, including:

[0087] Locating the image of the vehicle window in the side image, determining the state of the vehicle window according to the image of the vehicle window, if the vehicle window is in an open state, stopping selecting the side image of the vehicle from the target video, and if the vehicle window is in a closed state, continuing to select the side image of the vehicle from the target video until it is determined that the vehicle window is in an open state;

[0088] A specific frame is selected as the side image of the vehicle, and its expression is:

[0089] ;

[0090] in, Indicates Frame image, in a video stream, each frame is a snapshot at a point in time. Refers to the video frames selected at a certain time interval. Represents a single frame in a video stream, Indicates the frame number. Represents the time interval, which is the time difference between two consecutive frames and is used to define the frequency of selecting frames.

[0091] After acquiring the image of the side of the vehicle, the image of the vehicle window can be directly located in the image of the side of the vehicle.

[0092] In addition, the four cameras in the vehicle intelligent security inspection system shoot the four windows of the vehicle respectively, and the four cameras can transmit the captured videos to the controller in real time, and the controller can obtain the video of the window in real time, that is, the controller can collect the target video of the vehicle, and the target video has the image of the side of the vehicle, so that the controller can select at least one image of the side of the vehicle from the target video, that is, when the controller collects the target video of the vehicle in real time, it selects at least one image of the side of the vehicle from the target video in real time. Once the controller selects the image of the side of the vehicle, the image of the window of the vehicle can be located in the image of the side, and the state of the window of the vehicle is determined according to the image of the window of the vehicle. If it is determined that the window is in the open state, it means that the window of the vehicle has been opened, and there is no need to continue to determine the state of the window, that is, it can stop selecting the image of the side of the vehicle from the target video; if the window is in the closed state, it is necessary to continue to select the image of the side of the vehicle from the target video, and continue to determine the state of the window through the image of the side of the vehicle until the window is in the open state, and then stop selecting the image of the side of the vehicle from the target video to determine the state of the window.

[0093] It should be noted that when selecting an image of the side of a vehicle from a target video, you can first select an image of the side of a vehicle and determine the state of the window. If it is determined that the window is in an open state, stop selecting an image of the side of the vehicle from the target video. If it is determined that the window is in a closed state, continue to select an image of the side of a vehicle from the target video until it is determined that the window is in an open state. Of course, when selecting an image of the side of a vehicle from a target video, you can also select two images of the side of the vehicle each time, and determine the state of the window based on these two images of the side of the vehicle. The specific number of images of the side of the vehicle selected from the target video is not limited in the embodiments of the present application.

[0094] In addition, in some implementations, the implementation method of selecting at least one image of the side of the vehicle from the target video may be: selecting the image of the side of the vehicle from the target video at a preset time interval.

[0095] Among them, when the controller captures the target video of the vehicle in real time, the target video of the vehicle includes multiple images of the side of the vehicle. Therefore, each time the image of the side of the vehicle is captured, the image of the side of the vehicle can be selected from the target video according to a preset time interval, which can ensure that the state of the window can be determined more accurately based on the image of the side of the vehicle.

[0096] It should be noted that the preset time interval can be set according to actual needs, for example, the preset time interval is 100 milliseconds, and for another example, the preset time interval is 200 milliseconds. This is not limited in the embodiments of the present application.

[0097] In addition, in some implementations, when detecting the license plate on the vehicle, before detecting the state of the vehicle window, the vehicle window detection method may further include: acquiring images of the side bodies of a plurality of sample vehicles, the images of the side bodies of the sample vehicles including sample images of vehicle windows of the sample vehicles, the sample images of the vehicle windows including images of the vehicle windows in an open state and images of the vehicle windows in a closed state;

[0098] Extracting features of the window glass from sample images of the windows of sample vehicles, including the proportion of the window glass in the image, the shape of the window, and the clarity of the edge, to identify the outline of the window;

[0099] Based on multiple sample vehicle side images, a convolutional neural network is used to establish a window recognition model. The window recognition model is used to identify the state of the window. During the training process, the marked sample images are used to learn the features of the open and closed states.

[0100] Determining the state of the vehicle window based on the image of the vehicle window includes:

[0101] According to the vehicle window recognition model, the image of the selected vehicle window is recognized to determine the state of the vehicle window;

[0102] Perform anomaly detection, analyze the occlusion or damage of the window, and determine the corresponding response mechanism, including: analyzing the texture and color characteristics of the window area to determine whether there is an object blocking the window. If occlusion is detected, further determine whether an alarm or manual intervention is needed based on the degree of occlusion and the characteristics of the obstruction. Check whether the edges of the window area are intact and whether there are signs of cracks, breakage or damage. If damage is detected, record and report it for further manual verification or processing.

[0103] It should be noted that the process of abnormal detection and analysis of the obstruction or damage of the car window includes:

[0104] The collected images are grayed, filtered and denoised to improve the accuracy of subsequent processing. The texture features are extracted using the gray-level co-occurrence matrix, which is expressed as:

[0105] ;

[0106] in, Indicated in angle Next, grayscale and The frequency of occurrence, Represents the total number of pixel pairs, represents a function of pixel pairs, Indicates the position in the image The pixel gray value, To determine the location and Whether the pixel pair meets the gray level condition and ;

[0107] Extract color histogram in color space, extract color features, and analyze the color distribution of the image;

[0108] Compare the texture and color features of the current image with the baseline features of a normal car window and use a statistical test to detect occlusion, which is expressed as:

[0109] ;

[0110] in, Represents the chi-square statistic, which is used to compare actual observations and expected value The difference between represents the actual observed value, represents the expected value under normal conditions;

[0111] Calculate the ratio of the blocked area to the total window area to assess the degree of blockage;

[0112] The Canny edge detection algorithm is used to identify the integrity of the window edge. Its expression is:

[0113] ;

[0114] in, Indicates at location The edge strength, and Respectively indicate the location The Sobel operator calculation results in the horizontal and vertical directions at are used for edge detection. Represents a small positive number, used to avoid division by zero during calculations;

[0115] Cracks and breakages are identified by analyzing the discontinuity and fragmentation of the edges, and the expression is:

[0116] ;

[0117] in, Represents the crack score, which is used to quantify the degree of cracks or fragmentation in the image. Indicates the total number of cracked or broken areas detected, Indicates The characteristics of the crack or broken area, Represents the cumulative characteristics of all cracked or broken areas;

[0118] Determine whether to issue an alarm or perform manual intervention based on the degree of obstruction and crack characteristics;

[0119] Among them, multiple images of the side of the sample vehicles can be obtained in advance, which is equivalent to obtaining multiple training samples, and then a window recognition model is established based on the images of the side of the sample vehicles, which is equivalent to using multiple training samples to train the model to obtain a window recognition model that can recognize the state of the window. The image of the side of the sample vehicle includes a sample image of the window, and the sample image of the window includes an image of the window in an open state and an image of the window in a closed state, so that the window recognition model constructed based on the sample image of the window can accurately recognize the state of the window. Then, the image of the window of the selected vehicle can be identified based on the window recognition model, and the state of the window of the vehicle can be determined. Moreover, through the window recognition model, the image of the window of the selected vehicle can be quickly identified, and the efficiency of determining the state of the window can be improved.

[0120] It should be noted that when establishing a vehicle window recognition model through the image of the side of the sample vehicle, the vehicle window recognition model can be obtained by training with reference to the existing image training method. The vehicle window recognition model can be a long short-term memory network model, a CNN model, etc.

[0121] In addition, in some implementations, the state of the vehicle windows can be determined based on the images of the vehicle windows by: constructing virtual coordinates corresponding to the vehicle based on the images of the selected vehicle windows, the virtual coordinates including the coordinates of the vehicle body and the coordinates of the windows; determining a target ratio of the coordinates of the window to the coordinates of the vehicle body; if the target ratio is greater than or equal to a preset threshold, determining that the vehicle windows are in an open state; if the target ratio is less than the preset threshold, determining that the vehicle windows are in a closed state.

[0122] Among them, after obtaining the image of the vehicle window, the virtual coordinates corresponding to the vehicle can be constructed based on the image of the selected vehicle window, that is, the coordinate system of the image of the vehicle window is constructed, and the image of the vehicle window corresponds to the vehicle, so that the virtual coordinates correspond to the actual coordinates of the vehicle, and the virtual coordinates include the coordinates of the vehicle body and the coordinates of the window. After that, the target ratio of the coordinates of the window to the coordinates of the body can be determined. The window refers to the glass in the vehicle window, that is, the target ratio of the coordinates of the window glass to the coordinates of the body is determined. Among them, when the glass of the window is in the open state, the ratio of the coordinates of the window glass to the coordinates of the body is a first ratio, and when the window is in the closed state, the ratio of the coordinates of the window glass to the coordinates of the body is a second ratio, and the first ratio is different from the second ratio. Therefore, it is possible to determine whether the window is open by determining the size of the target ratio.

[0123] In addition, the preset threshold value may be the ratio of the coordinates of the window glass to the coordinates of the vehicle body when the window glass is in the open state. If the target ratio is greater than or equal to the preset threshold value, it indicates that the window glass is already in the open state, and it is determined that the vehicle window is in the open state; if the target ratio is less than the preset threshold value, it indicates that the window glass is not opened, or is opened slightly, and it is determined that the window is in the closed state.

[0124] In addition, in some implementations, after acquiring the image of the side of the vehicle, the vehicle window drop detection method may further include: if a license plate other than the current vehicle is detected, terminating the detection of the state of the window of the current vehicle.

[0125] Among them, if a license plate other than the current vehicle is detected, it indicates that vehicles other than the current vehicle have entered the detection range of the vehicle intelligent security inspection system, and the current vehicle has been detected, so the detection of the status of the windows of the current vehicle can be terminated, thereby saving the power consumption of the vehicle intelligent security inspection system.

[0126] In addition, in the embodiment of the present application, the window being in an open state includes: the ratio of the area of ​​the window glass within the window frame of the vehicle to the area of ​​the window frame is less than or equal to one third, which is equivalent to the window glass having fallen by two-thirds, thereby preventing the window glass from blocking the occupants inside the vehicle and ensuring that the camera in the vehicle intelligent security inspection system can capture images of the occupants inside the vehicle.

[0127] In addition, in some implementations, the implementation method of determining the state of the vehicle window based on the image of the vehicle window may be: if a human figure is detected in the image of the vehicle window, the state of the vehicle window is marked as open.

[0128] Among them, once the vehicle enters the detection range of the vehicle intelligent security inspection system, the license plate capture camera of the vehicle intelligent security inspection system captures the vehicle's license plate, and the wide-angle video camera faces the vehicle window. The wide-angle video camera can not only shoot the window, but also shoot the occupants inside the vehicle, and the wide-angle video camera transmits the shot video to the controller. After the controller obtains the video, once the controller detects the human image inside the vehicle through the image of the window in the video, indicating that the vehicle's window is already in an open state, the state of the vehicle's window is marked as open.

[0129] Apply the human figure detection algorithm to search for human figures in the image. If a human figure is detected in the image of the car window, proceed to the next step of analysis.

[0130] Extract the candidate human figure regions in the car window image, and analyze the shape descriptor of each candidate human figure region by combining the boundary length and region area;

[0131] The human figure recognition model is trained through the convolutional neural network model. The human figure recognition model is applied to each screened human figure candidate area to calculate the probability that it belongs to a human figure. The expression is:

[0132] ;

[0133] in, Represents a given feature vector When , the probability that the candidate region belongs to a human shape is, The feature vector representing the candidate region contains various features used for human recognition. Represents the activation function, which is used in the output layer of the classification model to convert the linear output into a probability value. Represents the weight vector of the model, indicating the contribution of the feature to the classification result. Represents the bias term, which is used to control the offset of the classification threshold;

[0134] According to the output probability of the human figure recognition model, combined with the preset threshold, it is determined whether a human figure image is detected, and then the window status is comprehensively inferred. The expression is:

[0135] ;

[0136] ;

[0137] in, Represents the comprehensive evaluation value, which is calculated by The single value calculated and summed is used to evaluate the state of the window and determine whether the window is open. Indicates the position in the image The strength of the humanoid features at Represents the binary representation of the window area. If the position In the window area, is equal to 1; otherwise, it is equal to 0, which is used to limit the calculation to the window area. Indicates location The shortest distance to the nearest window edge is used to evaluate whether the detected human figure is close to the window edge. Represents the threshold of the distance between the human figure and the window edge. If Less than this threshold , it is believed that the figure is likely outside the car window, so the car window may be open. Represents a very small positive number to prevent the denominator from being zero. Represents an integer greater than 1, used to magnify the distance The impact of the results, An image representing the entire analysis, represents the coordinates of any position in the image, represents the set of all edge pixels, Indicates the coordinates of edge pixels, indicating the edge position of the window, Indicates in the collection Find the edge pixel that minimizes the distance formula. It should be noted that when the detected human feature has a high intensity and is located in the window area and close to the window edge, The value of will be relatively large because the humanoid feature strength and will contribute a larger value. On the contrary, if the human feature is weak or far away from the edge of the window, The value of will be smaller;

[0138] If it is inferred that a human figure is detected in the image of the vehicle window, the state of the vehicle window is marked as an open state.

[0139] In addition, in some implementations, the vehicle window drop detection method may further include: if the vehicle window is in an open state, collecting a corresponding facial portrait inside the vehicle; if the vehicle window is in a closed state, issuing a prompt message, the prompt message being used to instruct the vehicle window to switch from a closed state to an open state until the vehicle window is in an open state.

[0140] Among them, if the car window is in the open state, at this time, the car window will not block the people inside the vehicle, so that the camera in the vehicle intelligent security inspection system can capture the facial portraits of the people inside the vehicle and send the captured images to the controller, and the controller can collect the facial portraits of the people inside the vehicle.

[0141] In addition, the vehicle intelligent security inspection system needs to detect the occupants inside the vehicle. If the window is in a closed state, a prompt message is issued. The prompt message can be a buzzer, a voice prompt, or other prompt messages. After the occupants inside the vehicle receive the prompt message, they can open the vehicle window so that the vehicle intelligent security inspection system can detect the occupants inside the vehicle. If the window is still in a closed state after the prompt message is issued, the prompt message will continue to be issued until the window is opened.

[0142] In addition, in the embodiments of the present application, Figure 2 The vehicle window drop detection method is described as follows:

[0143] When the license plate on the vehicle is detected, an image of the side of the vehicle is acquired, and then an image of the window is acquired based on the image of the side of the vehicle, and the state of the window is determined according to the image of the window. Specifically, after acquiring the image of the window, if there is a human-shaped image in the image of the window, it is determined that the window is in an open state, that is, the window has fallen down. If the ratio of the area of ​​the glass of the window in the window frame of the vehicle to the area of ​​the window frame is less than or equal to one third, that is, the window has fallen down by more than two thirds, it is determined that the window has fallen down. That is, when acquiring the image of the window, it is only necessary to determine whether there is a human-shaped image in the image of the window and to determine that the window has fallen down by more than two thirds. Only one of the two is satisfied to determine that the window has fallen down. In addition, the vehicle window falling detection method may also include: when determining that the window is in a closed state, reacquiring an image of the side of the vehicle until the window is determined to be in an open state through the image of the side of the vehicle.

[0144] Among them, when it is determined that the ratio of the area of ​​the window glass within the window frame of the vehicle to the area of ​​the window frame is less than or equal to one third, it can be determined according to the window recognition model. Of course, it can also be determined according to the target ratio of the coordinates of the window to the coordinates of the vehicle body. For details, please refer to the method in the above embodiment, which will not be repeated here.

[0145] In addition, in the embodiments of the present application, Figure 2 The application of the vehicle window drop detection method to the vehicle intelligent security inspection system is explained:

[0146] When the controller in the vehicle intelligent security inspection system detects the license plate on the vehicle, it obtains the image of the side of the vehicle, and the controller can be in the initial state, and the flag FlagLf1 is 0, and the flag FlagLf2 is 0. Then the controller obtains the image of the window based on the image of the side of the vehicle, and determines the state of the window according to the image of the window. Specifically, after obtaining the image of the window, if the controller determines that there is a human image in the image of the window, it determines that the window is in the open state, that is, the window has been lowered, and the flag FlagLf1 is 1; if the controller determines from the image of the window that the ratio of the area of ​​the window glass in the window frame of the vehicle to the area of ​​the window frame is less than or equal to one third, that is, the window has fallen by more than two thirds, then it determines that the window has been lowered, and the flag FlagLf2 is 1. The controller only needs to determine whether FlagLf2 is 1 or FlagLf1 is 1 to determine whether the window is down. That is, the controller obtains the image of the window and only needs to determine whether there is a human image in the image of the window and whether the window has fallen by more than two-thirds. Only one of these two conditions needs to be met to determine that the window has been lowered.

[0147] In addition, in the embodiment of the present application, after acquiring the image of the vehicle window, it is detected whether the vehicle window is lowered, and it is detected whether there is a human figure image in the image of the vehicle window. This is because it is difficult to detect whether the vehicle window is lowered for transparent window glass, and relying on the human figure image to determine whether the vehicle window is lowered can more accurately determine the state of the vehicle window.

[0148] In the embodiment of the present application, when the license plate on the vehicle is detected, an image of the side of the vehicle is obtained; based on the image of the side, an image of the window is obtained, and the state of the window is determined according to the image of the window, wherein the state of the window includes: the window is in an open state and the window is in a closed state. That is, in the embodiment of the present application, when the license plate on the vehicle is detected, the state of the window of the vehicle is detected, thereby avoiding manual detection of the window of the vehicle, and improving the efficiency of security inspection, thereby shortening the security inspection process and saving police resources.

[0149] An embodiment of the present application provides a vehicle intelligent security inspection system, a vehicle intelligent security inspection system, and a vehicle intelligent security inspection system. The vehicle intelligent security inspection system includes a controller and at least four cameras, the cameras are electrically connected to the controller, and the controller is used to execute the vehicle window drop detection method in any of the above embodiments.

[0150] The vehicle intelligent security inspection system may include five cameras, one of which is a license plate capture camera, and the other four are wide-angle video cameras. Once the vehicle drives to the vehicle intelligent security inspection system, the license plate acquisition camera can acquire the vehicle's license plate and send the acquired license plate to the controller. The four window acquisition cameras are respectively facing the four windows of the vehicle. After receiving the license plate information of the vehicle, the controller can control the four license plate acquisition cameras to respectively acquire images of the four windows of the vehicle, and send the acquired window images to the controller. The controller detects the status of the windows through the window images.

[0151] In addition, in an embodiment of the present application, in the vehicle intelligent security inspection system, the five cameras and the controller can be electrically connected through a switch, that is, the five cameras are connected to the switch through cables, and the controller is also connected to the switch through cables, so that data can be transmitted to each other between the five cameras and the controller.

[0152] The vehicle window drop detection method provided in the embodiment of the present application can be executed by a vehicle intelligent security inspection device. In the embodiment of the present application, the vehicle window drop detection method executed by the vehicle intelligent security inspection device is taken as an example to illustrate the vehicle intelligent security inspection device provided in the embodiment of the present application.

[0153] Reference Figure 3 , showing a schematic diagram of a vehicle intelligent security inspection device provided by an embodiment of the present application, such as Figure 3 As shown, the vehicle intelligent security inspection device 200 includes:

[0154] The detection module 201 is used to obtain a side image of the vehicle when a license plate on the vehicle is detected;

[0155] The determination module 202 is used to obtain an image of a vehicle window based on the side image, and determine the state of the vehicle window according to the image of the vehicle window, wherein the state of the vehicle window includes: the vehicle window is in an open state and the vehicle window is in a closed state.

[0156] Optionally, the detection module includes:

[0157] A first acquisition unit is used to acquire a target video of a vehicle, wherein the target video includes images of the side bodies of a plurality of vehicles, wherein the images of the side bodies of the vehicles include images of vehicle windows, and at least one image of the side body of a vehicle is selected from the target video;

[0158] Identify modules, including:

[0159] A determination unit is used to locate the image of the vehicle window in the side image, and determine the state of the window based on the image of the window. If the window is in an open state, stop selecting the side image of the vehicle from the target video; if the window is in a closed state, continue selecting the side image of the vehicle from the target video until it is determined that the window is in an open state.

[0160] Optionally, the first acquisition unit is further used for:

[0161] According to a preset time interval, multiple side images of the vehicle are selected from the target video.

[0162] Optionally, the vehicle intelligent security inspection device further includes:

[0163] An acquisition module, used to acquire images of the side bodies of a plurality of sample vehicles, wherein the images of the side bodies of the sample vehicles include sample images of vehicle windows of the sample vehicles, and the sample images of vehicle windows include images of vehicle windows in an open state and images of vehicle windows in a closed state;

[0164] A building module is used to build a vehicle window recognition model based on multiple sample vehicle side body images, and the vehicle window recognition model is used to identify the state of the vehicle window;

[0165] The determination unit is also used to:

[0166] According to the vehicle window recognition model, the image of the selected vehicle window is recognized to determine the state of the vehicle window.

[0167] Optionally, the determining unit is further configured to:

[0168] According to the image of the selected vehicle window, construct virtual coordinates corresponding to the vehicle, the virtual coordinates including the coordinates of the vehicle body and the coordinates of the vehicle window;

[0169] Determine a target ratio of the coordinates of the vehicle window to the coordinates of the vehicle body;

[0170] If the target ratio is greater than or equal to the preset threshold, it is determined that the vehicle's windows are in an open state; if the target ratio is less than the preset threshold, it is determined that the vehicle's windows are in a closed state.

[0171] Optionally, the vehicle intelligent security inspection device further includes:

[0172] The suspending module is used to suspend the detection of the state of the window of the current vehicle if a license plate other than the current vehicle is detected.

[0173] Optionally, the vehicle window being in an open state includes: a ratio of an area of ​​the vehicle window glass within a vehicle window frame to an area of ​​the vehicle window frame is less than or equal to one third.

[0174] Optionally, the vehicle intelligent security inspection device further includes:

[0175] The first marking module is used to mark the state of the vehicle window as an open state if a human figure image is detected in the image of the vehicle window.

[0176] Optionally, the vehicle intelligent security inspection device further includes:

[0177] A collection module, used to collect the corresponding face image inside the vehicle if the window is in an open state;

[0178] The prompt module is used to send a prompt message if the vehicle window is in a closed state, and the prompt message is used to instruct the vehicle window to switch from a closed state to an open state until the vehicle window is in an open state.

[0179] In the embodiment of the present application, when the license plate on the vehicle is detected, an image of the side of the vehicle is obtained; based on the image of the side, an image of the window is obtained, and the state of the window is determined according to the image of the window, wherein the state of the window includes: the window is in an open state and the window is in a closed state. That is, in the embodiment of the present application, when the license plate on the vehicle is detected, the state of the window of the vehicle is detected, thereby avoiding manual detection of the window of the vehicle, and improving the efficiency of security inspection, thereby shortening the security inspection process and saving police resources.

[0180] The vehicle intelligent security inspection device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0181] The vehicle intelligent security inspection device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0182] The vehicle intelligent security inspection device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0183] Alternatively, if Figure 4 As shown, an embodiment of the present application further provides an electronic device a300, including a processor a301 and a memory a302, wherein the memory a302 stores programs or instructions that can be executed on the processor a301, and when the program or instructions are executed by the processor a301, the various steps of the embodiment of the above-mentioned vehicle window drop detection method are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described again here.

[0184] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0185] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0186] The electronic device b1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory b1009, and a processor b1010 and other components.

[0187] Those skilled in the art will appreciate that the electronic device b1000 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor b1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0188] Among them, the processor b1010 is used to obtain an image of the side of the vehicle when a license plate on the vehicle is detected; based on the image of the side, an image of the vehicle window is obtained, and the state of the window is determined according to the image of the window, wherein the state of the window includes: the window is in an open state and the window is in a closed state.

[0189] In the embodiment of the present application, when the license plate on the vehicle is detected, an image of the side of the vehicle is obtained; based on the image of the side, an image of the vehicle window is obtained, and the state of the window is determined according to the image of the window. That is, in the embodiment of the present application, when the license plate on the vehicle is detected, the state of the vehicle window is detected, thereby avoiding manual detection of the vehicle window, and improving the efficiency of security inspection, thereby shortening the security inspection process and saving police resources.

[0190] Optionally, the processor b1010 is further configured to:

[0191] Capturing a target video of a vehicle, the target video comprising images of the side bodies of a plurality of vehicles, the images of the side bodies of the vehicles including images of vehicle windows, and selecting at least one image of the side body of a vehicle from the target video;

[0192] The processor b1010 is also used to:

[0193] The image of the vehicle's window is located in the side image, and the state of the window is determined based on the image of the window. If the window is in an open state, the selection of the vehicle's side image from the target video is stopped; if the window is in a closed state, the selection of the vehicle's side image from the target video is continued until it is determined that the window is in an open state.

[0194] Optionally, the processor b1010 is further configured to:

[0195] According to the preset time interval, the image of the side of the vehicle is selected from the target video.

[0196] Optionally, the processor b1010 is further configured to:

[0197] Acquire multiple images of the side of a sample vehicle, wherein the images of the side of the sample vehicle include sample images of vehicle windows of the sample vehicle, and the sample images of the vehicle windows include images of the vehicle windows in an open state and images of the vehicle windows in a closed state;

[0198] A vehicle window recognition model is established based on the side images of multiple sample vehicles, and the vehicle window recognition model is used to identify the state of the vehicle window;

[0199] The processor b1010 is also used to:

[0200] According to the vehicle window recognition model, the image of the side of the selected vehicle is recognized to determine the state of the vehicle window.

[0201] Optionally, the processor b1010 is further configured to:

[0202] According to the image of the vehicle window of the selected vehicle, construct virtual coordinates corresponding to the vehicle, the virtual coordinates including the coordinates of the vehicle body and the coordinates of the vehicle window;

[0203] Determine a target ratio of the coordinates of the vehicle window to the coordinates of the vehicle body;

[0204] If the target ratio is greater than or equal to the preset threshold, it is determined that the vehicle's windows are in an open state; if the target ratio is less than the preset threshold, it is determined that the vehicle's windows are in a closed state.

[0205] Optionally, the processor b1010 is further configured to:

[0206] If a license plate other than the current vehicle is detected, the detection of the state of the window of the current vehicle is stopped.

[0207] Optionally, the vehicle window being in an open state includes: a ratio of an area of ​​the vehicle window glass within a vehicle window frame to an area of ​​the vehicle window frame is less than or equal to one third.

[0208] Optionally, the processor b1010 is further configured to:

[0209] If a human figure image is detected in the image of the vehicle window, the state of the vehicle window is marked as an open state.

[0210] Optionally, the processor b1010 is further configured to:

[0211] If the car window is open, the face image inside the vehicle is collected;

[0212] If the window is in the closed state, a prompt message is issued, and the prompt message is used to instruct the vehicle window to switch from the closed state to the open state until the window is in the open state.

[0213] In the embodiment of the present application, when the license plate on the vehicle is detected, an image of the side of the vehicle is obtained; based on the image of the side, an image of the vehicle window is obtained, and the state of the window is determined according to the image of the window, wherein the state of the window includes: the window is in an open state and the window is in a closed state. That is, in the embodiment of the present application, when the license plate on the vehicle is detected, the state of the vehicle window is detected, thereby avoiding manual detection of the vehicle window, and thus improving the efficiency of security inspection, thereby shortening the security inspection process and saving police resources.

[0214] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch analysis device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0215] The memory b1009 can be used to store software programs and various data. The memory b1009 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory b1009 can include a volatile memory or a non-volatile memory, or the memory x09 can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory b1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0216] The processor b1010 may include one or more processing units; optionally, the processor b1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor b1010.

[0217] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned vehicle window drop detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0218] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0219] An embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned vehicle window drop detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0220] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0221] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned vehicle window drop detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0222] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0223] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0224] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A vehicle window drop detection method, characterized in that: Applied to a vehicle intelligent security inspection system, the vehicle intelligent security inspection system includes at least four cameras, the vehicle intelligent security inspection system is used to perform security inspection on the vehicle, the vehicle includes four windows, each of the windows is photographed by one of the cameras, and the vehicle window fall detection method includes: When a license plate on a vehicle is detected in the security inspection area, an image of the side of the vehicle is acquired; Based on the image of the side of the vehicle, a vehicle window recognition model is established, and the state of the vehicle window is determined according to the image of the vehicle window and the vehicle window recognition model, wherein the state of the vehicle window includes: the vehicle window is in an open state and the vehicle window is in a closed state; The determining the state of the vehicle window according to the image of the vehicle window comprises: Applying a human figure detection algorithm to search for a human figure in the image, if a human figure image is detected in the image of the vehicle window, proceeding to the next step of analysis; Extracting candidate human figure regions in the image of the vehicle window, and analyzing the shape descriptor of each candidate human figure region in combination with the boundary length and the region area; The human figure recognition model is trained through the convolutional neural network model. The human figure recognition model is applied to each screened human figure candidate area to calculate the probability that it belongs to a human figure. The expression is: ； in, Represents a given feature vector When , the probability that the candidate region belongs to a human shape is, The feature vector representing the candidate region contains various features used for human recognition. Represents the activation function, which is used in the output layer of the classification model to convert the linear output into a probability value. Represents the weight vector of the model, indicating the contribution of the feature to the classification result. Represents the bias term, which is used to control the offset of the classification threshold; According to the output probability of the human figure recognition model, combined with the preset threshold, it is determined whether a human figure image is detected, and then the window status is comprehensively inferred. The expression is: ； ； in, Represents the comprehensive evaluation value, which is calculated by The single value calculated and summed is used to evaluate the state of the window and determine whether the window is open. Indicates the position in the image The strength of the humanoid features at represents the binary representation of the window area, Indicates location The shortest distance to the nearest window edge is used to evaluate whether the detected human figure is close to the window edge. The threshold value representing the distance between the human figure and the edge of the car window, Represents a positive number, used to prevent the denominator from being zero. Represents an integer greater than 1, used to enlarge The impact of the results, An image representing the entire analysis, represents the coordinates of any position in the image, represents the set of all edge pixels, Indicates the coordinates of edge pixels, indicating the edge position of the window; If it is inferred that a human figure is detected in the image of the vehicle window, the state of the vehicle window is marked as an open state.

2. A vehicle window drop detection method according to claim 1, characterized in that: The step of acquiring the side image of the vehicle comprises: Capturing a target video when the vehicle passes through a security inspection area, the target video includes multiple side images of the vehicle, the side images of the vehicle include multiple frames of images of vehicle windows of the vehicle, and selecting at least one frame including a side image of the target vehicle from the target video, and each side image of the target vehicle includes a vehicle window portion; At a preset time interval, selecting a specific frame as a side image of the vehicle; Processing the selected vehicle side image, and locating the vehicle window area from the vehicle side image using image segmentation technology; Extracting the outline, area and shape features of the vehicle window from the located vehicle window area; Based on the side image, acquiring an image of a window of the vehicle, and determining a state of the window according to the image of the window, comprising: Locating the image of the vehicle window in the side image, determining the state of the vehicle window according to the image of the vehicle window, and if the vehicle window is in an open state, stopping selecting the side image of the vehicle from the target video, and if the vehicle window is in a closed state, continuing to select the side image of the vehicle from the target video until it is determined that the vehicle window is in an open state; The expression for selecting a specific frame as the side image of the vehicle is: ； in, Indicates Frame image, in a video stream, each frame is a snapshot at a point in time. Refers to the video frames selected at a certain time interval. Represents a single frame in a video stream, Indicates the frame number. Represents the time interval, which is the time difference between two consecutive frames and is used to define the frequency of selecting frames.

3. A vehicle window drop detection method according to claim 2, characterized in that: When detecting the license plate on the vehicle, before acquiring the image of the side of the vehicle, the vehicle window drop detection method further includes: Acquire multiple images of the side of a sample vehicle, wherein the images of the side of the sample vehicle include sample images of vehicle windows of the sample vehicle, and the sample images of the vehicle windows include images of the vehicle windows in an open state and images of the vehicle windows in a closed state; Extracting features of the window glass from the sample image of the window of the sample vehicle, including the proportion of the window glass in the image, the shape of the window, and the clarity of the edge, so as to identify the outline of the window; Based on the images of the side of the plurality of sample vehicles, a convolutional neural network is used to establish a vehicle window recognition model, wherein the vehicle window recognition model is used to recognize the state of the vehicle window, and during the training process, the marked sample images are used to learn the features of the open and closed states; The determining the state of the vehicle window according to the image of the vehicle window comprises: Recognize the image of the selected vehicle window according to the vehicle window recognition model, and determine the state of the vehicle window; Perform anomaly detection, analyze the occlusion or damage of the window, and determine the corresponding response mechanism, including: analyzing the texture and color characteristics of the window area to determine whether there is an object blocking the window. If occlusion is detected, further determine whether an alarm or manual intervention is needed based on the degree of occlusion and the characteristics of the obstruction. Check whether the edge of the window area is intact and whether there are signs of cracks, breakage or damage. If damage is detected, record and report it, and further manual verification or processing.

4. A vehicle window drop detection method according to claim 2, characterized in that: The determining the state of the vehicle window according to the image of the vehicle window comprises: constructing virtual coordinates corresponding to the vehicle according to the selected image of the vehicle window, the virtual coordinates including coordinates of the vehicle body and coordinates of the vehicle window; Determine a target ratio of the coordinates of the vehicle window to the coordinates of the vehicle body, and set a threshold to analyze the relationship between the target ratio and the threshold; If the target ratio is greater than or equal to a preset threshold, it is determined that the window of the vehicle is in an open state; if the target ratio is less than the preset threshold, it is determined that the window of the vehicle is in a closed state.

5. A vehicle window drop detection method according to claim 3, characterized in that: The process of abnormality detection and analyzing the obstruction or damage of the vehicle window includes: The collected image is grayed, filtered and denoised, and the texture features are extracted using the gray-level co-occurrence matrix, which is expressed as: ； in, Indicated in angle Next, grayscale and The frequency of occurrence, Represents the total number of pixel pairs, represents a function of pixel pairs, Indicates the position in the image The pixel gray value, To determine the location and Whether the pixel pair meets the gray level condition and ; Extract color histogram in color space, extract color features, and analyze the color distribution of the image; Compare the texture and color features of the current image with the baseline features of a normal car window and use a statistical test to detect occlusion, which is expressed as: ； in, Represents the chi-square statistic, which is used to compare actual observations and expected value The difference between represents the actual observed value, represents the expected value under normal conditions; Calculate the ratio of the blocked area to the total window area to assess the degree of blockage; The Canny edge detection algorithm is used to identify the integrity of the window edge. The expression is: ； in, Indicates at location The edge strength, and Respectively indicate the location The Sobel operator calculation results in the horizontal and vertical directions at are used for edge detection. Represents a small positive number, used to avoid division by zero during calculations; Cracks and breakages are identified by analyzing the discontinuity and fragmentation of the edges, and the expression is: ； in, Represents the crack score, which is used to quantify the degree of cracks or fragmentation in the image. Indicates the total number of cracked or broken areas detected, Indicates The characteristics of the crack or broken area, Represents the cumulative characteristics of all cracked or broken areas; Determine whether to issue an alarm or perform manual intervention based on the degree of obstruction and crack characteristics.

6. A vehicle window drop detection method according to claim 1, characterized in that: After acquiring the image of the side of the vehicle, the vehicle window drop detection method further includes: If a license plate other than the current vehicle is detected, the detection of the state of the window of the current vehicle is stopped.

7. A vehicle window drop detection method according to claim 1, characterized in that: The vehicle window being in an open state includes: a ratio of an area of ​​the glass of the vehicle window within the vehicle window frame to an area of ​​the vehicle window frame is less than or equal to one third.

8. A vehicle window drop detection method according to claim 1, characterized in that: The vehicle window drop detection method further includes: If the vehicle window is in an open state, a corresponding face image of the person inside the vehicle is collected; If the vehicle window is in a closed state, a prompt message is issued, wherein the prompt message is used to instruct the vehicle window to switch from a closed state to an open state until the vehicle window is in an open state.

9. A vehicle intelligent security inspection system, characterized in that: The vehicle intelligent security inspection system includes a controller and at least four cameras, the cameras are electrically connected to the controller, and the controller is used to execute the vehicle window drop detection method described in any one of claims 1-8.

10. A vehicle intelligent security inspection device, characterized in that: A detection module is included, which is used to obtain an image of the side of the vehicle when a license plate on the vehicle is detected; A determination module is used to obtain an image of a window of the vehicle based on the image of the side body; and to perform the vehicle window drop detection method according to any one of claims 1 to 8 according to the image of the window to determine the state of the window.

Images