Vehicle abnormal vibration detection method and device, electronic equipment and vehicle

Abstract

The invention provides a vehicle abnormal vibration detection method and device, electronic equipment and a vehicle, and the method comprises the steps: obtaining continuous multi-frame images sent by a plurality of vehicle-mounted cameras, and analyzing the image displacement of two adjacent frames of images of each vehicle-mounted camera; if the image displacement of the vehicle-mounted camera is greater than a set threshold value, determining the vehicle-mounted camera as an abnormal vibration camera, taking an image sent by the abnormal vibration camera as an abnormal vibration point image, and extracting key information of a target object in the abnormal vibration point image; and acquiring integrated information of the key information of the target object and the corresponding abnormal vibration point image as abnormal vibration detection information. According to the scheme, when the abnormal vibration of the vehicle is detected, the image which is shot by the vehicle-mounted camera corresponding to the abnormal vibration point and records the key information of the target object and the target object are integrated into the abnormal vibration detection information, and the abnormal vibration detection information is used as the support information for searching the personnel in charge subsequently; and loss caused by incapability of determining responsible personnel information is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle safety detection systems, and in particular to a vehicle abnormal vibration detection method, device, electronic equipment and vehicle. Background Art

[0002] At present, when vehicles are parked in some unattended or remote places, there may be a risk of being scratched or collided. The current detection method is to install sensors on the vehicle to sense whether there is a large vibration. When the sensor detects that the vehicle has a large vibration, the vehicle may be at risk of collision, scratching, etc., and the system will sound an alarm. Specifically, the sensor installed on the vehicle detects the vibration amplitude of the vehicle and sends the detection result to the control unit of the vehicle. The control unit compares the vibration amplitude when the vehicle is in a normal parking state with the vibration amplitude sent by the sensor. When the difference is too large, it is determined that the vehicle has abnormal vibration and an alarm is issued. The above scheme can only provide a warning when a collision and scratch occur, but the actual application scenario may have the following problems: after a collision or scratch occurs, the responsible person leaves the scene, and it is difficult to determine the responsible person, which affects the subsequent processing process and causes irreparable losses. Summary of the invention

[0003] The technical problem to be solved by the present application is that it is difficult to determine the person responsible for the loss caused by an accidental collision or scratch of a vehicle in the prior art, and thus a method, device, electronic equipment and vehicle for detecting abnormal vibration of a vehicle are provided.

[0004] In a first aspect, the technical solution of the present application provides a method for detecting abnormal vibration of a vehicle, comprising:

[0005] Acquire continuous multi-frame images sent by multiple vehicle-mounted cameras, where different vehicle-mounted cameras are used to capture different directions of the environment where the vehicle is located;

[0006] Obtaining the image displacement of two adjacent frames of images of each of the vehicle-mounted cameras, and if the image displacement is greater than a set threshold, determining the corresponding vehicle-mounted camera as an abnormal vibration camera;

[0007] Acquire the image sent by the abnormal vibration camera as an abnormal vibration point image, and identify key information of the target object in contact with the vehicle in the abnormal vibration point image;

[0008] The integrated information of the key information of the target object and the corresponding abnormal vibration point image is obtained as abnormal vibration detection information.

[0009] In some embodiments of the abnormal vibration detection method for a vehicle, the image displacement of two adjacent frames of images of each vehicle-mounted camera is obtained, and if the image displacement is greater than a set threshold, the corresponding vehicle-mounted camera is determined to be an abnormal vibration camera, and the set threshold is obtained in the following manner:

[0010] The average value or median value of the image displacements of all the vehicle-mounted cameras obtained at the same time is used as the set threshold.

[0011] In some embodiments of the vehicle abnormal vibration detection method, the key information of the target object in contact with the vehicle in the abnormal vibration point image is identified, including:

[0012] Identifying the contour line of the vehicle in the abnormal vibration point image;

[0013] Acquiring the contour line of the environmental target in the abnormal vibration point image;

[0014] If the contour line of any of the environmental objects has an intersection with the contour line of the own vehicle, the environmental object is taken as the target object.

[0015] In some embodiments of the vehicle abnormal vibration detection method, the key information of the target object in contact with the vehicle in the abnormal vibration point image is identified, including:

[0016] If the target object is a human object, the facial information of the human object is identified as the key information; or, if the target object is a vehicle outline object, the license plate information of the vehicle outline object is identified as the key information.

[0017] In some embodiments of the vehicle abnormal vibration detection method, the step of obtaining the image displacement of two adjacent frames of images of each vehicle-mounted camera includes:

[0018] Determine the moving direction and moving amount of each pixel in two adjacent frames of images;

[0019] All pixels are grouped, and the pixels in each group have the same moving direction;

[0020] A target pixel group having the most pixels is determined, and the movement amounts of the pixels in the target pixel group are merged to obtain the image displacement amount.

[0021] The vehicle abnormal vibration detection method described in some embodiments further includes the following steps:

[0022] After associating the key information of the target object with the abnormal vibration point image as abnormal vibration detection information, the method further includes:

[0023] Acquiring the number of abnormal vibration point images;

[0024] Determine the duration of abnormal vibration based on the interval time between two adjacent frames of images sent by the vehicle-mounted camera and the number of abnormal vibration point images;

[0025] If the abnormal vibration lasts longer than the vibration duration threshold, it is determined that the vehicle is at risk of being stolen and an alarm signal is issued.

[0026] In a second aspect, the technical solution of the present application provides a vehicle abnormal vibration detection device, comprising:

[0027] An image acquisition module is configured to acquire continuous multi-frame images sent by a plurality of vehicle-mounted cameras, wherein different vehicle-mounted cameras are used to capture images in different directions of the environment where the vehicle is located;

[0028] an abnormal camera determination module, configured to obtain an image displacement of two adjacent frames of images of each of the vehicle-mounted cameras, and if the image displacement is greater than a set threshold, determine the corresponding vehicle-mounted camera as an abnormal vibration camera;

[0029] an abnormal image determination module, configured to obtain the image sent by the abnormal vibration camera as an abnormal vibration point image, and identify key information of a target object in contact with the vehicle in the abnormal vibration point image;

[0030] The abnormal vibration determination module is configured to obtain integrated information of the key information of the target object and the corresponding abnormal vibration point image as abnormal vibration detection information.

[0031] In a third aspect, the technical solution of the present application provides a computer-readable storage medium, in which program information is stored. After a computer reads the program information, the vehicle abnormal vibration detection method described in any one of the first aspects is executed.

[0032] In a fourth aspect, the technical solution of the present application provides an electronic device, comprising at least one processor and at least one memory, wherein at least one of the memories stores program information, and after reading the program information, at least one of the processors executes the vehicle abnormal vibration detection method described in any one of the first aspects.

[0033] In a fifth aspect, the technical solution of the present application provides a vehicle, in which the vehicle abnormal vibration detection device described in the second aspect or the computer-readable storage medium described in the third aspect or the electronic device described in the fourth aspect is configured.

[0034] The above technical solution has the following beneficial effects:

[0035] The vehicle abnormal vibration detection method, device, electronic device and vehicle provided by the present solution determine the abnormal vibration point of the vehicle and obtain abnormal vibration detection information by analyzing the images captured by the vehicle-mounted cameras in each direction of the existing surround view cameras. Specifically, the present application solution obtains continuous multiple frames of images sent by multiple vehicle-mounted cameras, analyzes the image displacement of two adjacent frames of images of each vehicle-mounted camera, and determines it as an abnormal vibration camera if the image displacement of the vehicle-mounted camera is greater than the set threshold. The image sent by the abnormal vibration camera is used as the abnormal vibration point image, and the key information of the target object in the abnormal vibration point image is extracted, and the key information of the target object and the corresponding abnormal vibration point image are obtained as the integrated information of the abnormal vibration detection information. The above scheme of the present application can determine the vehicle-mounted camera corresponding to the abnormal vibration point when abnormal vibration is detected in the vehicle, and integrate the image and the target object captured by the vehicle-mounted camera with the key information of the target object into abnormal vibration detection information as support information for subsequent search for responsible personnel, so as to avoid losses caused by the inability to clarify the information of the responsible personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a vehicle abnormal vibration detection method described in an embodiment of the present application;

[0037] Figure 2 This is a flow chart of the steps for obtaining the image displacement amount described in an embodiment of the present application;

[0038] Figure 3 This is a flow chart of a method for detecting abnormal vibration of a vehicle described in another embodiment of the present application;

[0039] Figure 4 This is a structural block diagram of a vehicle abnormal vibration detection device described in an embodiment of the present application;

[0040] Figure 5 Schematic diagram of the hardware connection relationship of an electronic device for executing a vehicle abnormal vibration detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] The specific implementation of the present application is further described below with reference to the accompanying drawings.

[0042] It is easy to understand that according to the technical solution of the present application, without changing the essential spirit of the present application, a variety of structural modes and implementation modes that can be replaced by those skilled in the art can be replaced with each other. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present application, and should not be regarded as the entirety of the present application or as a limitation or restriction on the technical solution of the application.

[0043] The directional terms such as up, down, left, right, front, back, front, back, top, bottom, etc. mentioned or may be mentioned in this specification are defined relative to the structures shown in the drawings. They are relative concepts and may change accordingly according to different positions and different usage conditions. Therefore, these or other directional terms should not be interpreted as restrictive terms.

[0044] The present application provides a method, device, electronic device and vehicle for detecting abnormal vibration of a vehicle. The method can be applied to a vehicle-mounted control terminal, such as a vehicle controller or other electronic control unit with data processing function installed on the vehicle, and the vehicle-mounted control terminal is connected to the vehicle's surround view camera in communication. The surround view camera itself includes multiple vehicle-mounted cameras, usually at least including vehicle-mounted cameras that can shoot the front, rear, left and right of the vehicle, and the surround view camera can shoot images within a 360° range of the environment in which the vehicle is located. In this solution, the vehicle-mounted control terminal can obtain images of different directions of the vehicle's environment captured by the surround view camera in real time. It can be understood that when this solution is applied to the field of vehicle safety technology, if the image captured by the surround view camera moves significantly under abnormal circumstances, it can be said that the surround view camera is vibrating, and the surround view camera is rigidly set on the vehicle. At this time, it can be determined that the vehicle may have risks such as collision or scratches.

[0045] The present application embodiment provides a method for detecting abnormal vibration of a vehicle, such as Figure 1 As shown, it can be applied to a vehicle-mounted control terminal, and the method includes:

[0046] S10: Acquire continuous multi-frame images sent by multiple vehicle-mounted cameras, where different vehicle-mounted cameras are used to capture images in different directions of the environment where the vehicle is located.

[0047] The present application scheme is applicable to vehicles equipped with surround-view cameras, which actually include multiple on-board cameras, and different on-board cameras are used to shoot different directions of the environment in which the vehicle is located. The on-board control terminal is connected to the surround-view camera of the vehicle in communication, so the on-board control terminal can obtain the image of the environment in which the vehicle is located captured by the surround-view camera in real time.

[0048] S20: Obtaining the image displacement of two adjacent frames of images of each of the vehicle-mounted cameras. If the image displacement is greater than a set threshold, the corresponding vehicle-mounted camera is determined to be an abnormal vibration camera.

[0049] This solution can determine the displacement of any frame of image compared to the previous frame of image by using correlation method, deep learning algorithm, etc. Assuming that the vehicle is equipped with four on-board cameras, which respectively take images of the vehicle in the front, rear, left and right directions, the image displacement of each on-board camera can be determined.

[0050] Under normal circumstances, the on-board camera is rigidly mounted on the vehicle, and if the vehicle is parked, the images captured by the on-board camera should also have relatively constant changes or remain unchanged. For example, if the vehicle is parked in a parking lot, there are other parked vehicles or some pedestrians in the parking lot, and the images captured by each camera may be of stationary vehicles or stationary or normally moving pedestrians, etc. Furthermore, if the vehicle is in motion, there will be no overall large movement between the continuous images captured instantly by the on-board camera, and only some of the pixels will move within the normal range.

[0051] S30: Acquire the image sent by the abnormal vibration camera as an abnormal vibration point image, and identify key information of the target object in contact with the vehicle in the abnormal vibration point image.

[0052] If the image captured by a certain vehicle camera moves greatly as a whole, it means that the vehicle camera has moved greatly. Since the vehicle camera is rigidly mounted on the vehicle, it can be considered that the vehicle has experienced abnormal vibration. At this time, when the image movement is large, the corresponding vehicle vibration amplitude is large, so the image of the corresponding abnormal vibration camera is selected for primary analysis, which can record the situation when the abnormal vibration occurs.

[0053] The target object may be a person, a vehicle, or a foreign object such as a stone, etc. When the target object contacts or collides with the vehicle, the vehicle will vibrate abnormally, and the abnormal vibration of the vehicle will drive the on-board camera to vibrate abnormally, resulting in a large displacement of the image captured by the on-board camera.

[0054] The key information may vary depending on the type of the target object. The target object may be identified through the key information. For example, if it is a vehicle, the key information may include the license plate number. Once the license plate number is determined, the vehicle may be easily identified.

[0055] S40: Obtaining integrated information of the key information of the target object and the corresponding abnormal vibration point image as abnormal vibration detection information, wherein the key information and the corresponding abnormal vibration point image can be processed according to the existing encoding method to obtain a complete data packet as integrated information, or the key information and the corresponding abnormal vibration point image can be stored in the same folder, and the compressed data packet obtained by compressing the folder can be used as integrated information, etc.

[0056] In this solution, when abnormal vibration occurs, key information of the target object causing the abnormal vibration can be recorded to facilitate subsequent search for the target object. In some solutions, the key information can also be sent to a target terminal, which can be a server, a user's mobile phone, etc., and the key information can be used as subsequent evidence information.

[0057] The above scheme provided by this embodiment determines the abnormal vibration point of the vehicle and obtains abnormal vibration detection information by analyzing the images taken by the on-board cameras in each direction of the existing surround view cameras. Specifically, the scheme of this application obtains continuous multiple frames of images sent by multiple on-board cameras, analyzes the image displacement of two adjacent frames of images of each on-board camera, and determines it as an abnormal vibration camera if the image displacement of the on-board camera is greater than the set threshold. The image sent by the abnormal vibration camera is used as the abnormal vibration point image, and the key information of the target object in the abnormal vibration point image is extracted. The integrated information of the key information of the target object and the corresponding abnormal vibration point image is obtained as abnormal vibration detection information. The above scheme of this application can determine the on-board camera corresponding to the abnormal vibration point when abnormal vibration is detected in the vehicle, and integrate the image and the target object recorded with the key information of the target object taken by the on-board camera into abnormal vibration detection information as support information for subsequent search for responsible personnel, thereby avoiding losses caused by the inability to clarify the information of the responsible personnel.

[0058] In the vehicle abnormal vibration detection method described in some schemes, in step S20, the set threshold is obtained in the following manner: the average value or median of the image displacement of all the vehicle-mounted cameras obtained at the same time is used as the set threshold. In practical applications, for the vehicle-mounted surround view camera, the all-round situation of the front, back, left, and right of the vehicle can be taken into account. The vehicle-mounted cameras set to monitor multiple directions, if the vibration location is to be determined, can be determined by comparing the displacement difference of the images captured by each vehicle-mounted camera, that is, the average value or median of the image displacement of all the vehicle-mounted cameras obtained at the same time can be used as the set threshold for comparison. For example, if the image displacement of one vehicle-mounted camera is much higher than the image displacement of other vehicle-mounted cameras, then the image displacement of the vehicle-mounted camera must be greater than the set threshold, indicating that the vibration source is in the shooting direction of this vehicle-mounted camera.

[0059] In some solutions, in step S20, for each image sent by the vehicle-mounted camera, the image displacement of two adjacent frames of the image of each vehicle-mounted camera is obtained by the following method:

[0060] S201: Identify at least one object block in each frame of image, where the object block includes at least one pixel.

[0061] An object block in an image is a location point or area including at least one pixel point, such as other vehicles, trees, etc. The image can be analyzed by a recognition algorithm such as artificial intelligence to analyze the various object blocks contained in the image.

[0062] S202: Obtain an object block displacement of at least one object block in two adjacent image frames.

[0063] The position of the object block in the image can be determined by the position of the pixels it occupies. For example, the pixels included in the object block are in the third to fifth rows and the seventh to ninth columns in the image, so the position of the object block in the image can be determined. Because the intervals between pixels in different rows and columns of the image can be determined by the resolution of the on-board camera. Therefore, after comparing two adjacent frames of images, it can be determined how many rows and columns the object block has moved in the row direction and how many columns it has moved in the column direction. Combined with the known row and column intervals, the displacement can be obtained. In order to simplify the calculation, the number of rows and columns moved can also be directly used as the displacement.

[0064] S203: The object block displacement amounts of the at least one object block are merged to obtain image displacement amounts of two adjacent frames of images.

[0065] The fusion method can be set. For example, if there are multiple object blocks in the image, the overall displacement of the image can be calculated by taking the average value of the displacement of each object block.

[0066] Under normal circumstances, if the image displacement between two adjacent frames of images taken by the vehicle camera is too large, it can be determined that the vehicle has experienced abnormal vibration, which may be caused by theft or collision. This solution can determine the image displacement by measuring the object block displacement of the object blocks in the continuous multiple frames of images taken by the vehicle camera. Since the vehicle camera is a component configured by the vehicle itself, the line connection between it and the various control modules on the vehicle has been completed, and there is no need to redesign and arrange the lines.

[0067] The step S201 of identifying at least one object block in each frame of the image includes:

[0068] S2011: Obtain the pixel value of each pixel point at each position in the image of each frame.

[0069] The pixel value is used to represent the value assigned by the processor when the original image is digitized, and can represent the average brightness information of the small square of pixel points in the original image, or the average reflection / transmission density information of the small square, etc. The relevant parts involving pixel values ​​in this application can use the same parameters, for example, all use brightness information to represent pixel values.

[0070] S2012: Obtain the difference between the pixel value of any pixel point at the current position and the pixel value of the pixel point at the adjacent position.

[0071] The difference can be obtained by directly subtracting the pixel values ​​of two pixels.

[0072] S2013: If the difference is less than the set pixel value, the pixel point at the current position and the pixel point at the adjacent position are classified as the same object block; otherwise, the pixel point at the current position is regarded as a boundary pixel point of the object block where the pixel point at the current position is located.

[0073] If the difference between the pixel values ​​of two adjacent pixels is not large, they can be considered to be the same object block, otherwise they are considered to be different object blocks. In practical application scenarios, it can be considered that the pixel values ​​of a certain object block should have a certain balance. For example, when the road surface in an image is used as an object block, the pixel values ​​should be approximately equal.

[0074] For image processing, obtaining the pixel value of each pixel in the image can be achieved by using a relatively simple algorithm. This method can simplify the calculation and improve the detection efficiency.

[0075] Furthermore, in step S202, obtaining the object block displacement of the at least one object block in the image of two adjacent frames includes:

[0076] S2021: Obtain first position information of each boundary pixel point of any object block in a previous frame image in the previous frame image.

[0077] After identifying each object block in the previous frame image, the first position information of each boundary pixel point of any object block in the previous frame image can be determined, which can be expressed in the form of row and column numbers where the pixel point is located.

[0078] S2022: Obtain second position information of each boundary pixel point of the same object block in the current frame image in the current frame image.

[0079] After identifying each object block in the current frame image, the second position information of each boundary pixel point of any object block in the current frame image can be determined, which can be expressed in the form of row and column numbers where the pixel point is located.

[0080] S2023: Obtain a change value between the second position information and the first position information as a displacement of the object block.

[0081] As mentioned above, because the intervals between pixels in different rows and columns of an image can be determined by the resolution of the onboard camera, by comparing two adjacent frames of images, it is possible to determine how many rows and columns the object block has moved in the row direction and how many columns it has moved in the column direction. Combining the known row and column intervals, the displacement can be obtained. To simplify the calculation, the number of rows and columns moved can also be used directly as the displacement.

[0082] Specifically, in step S2023, the change value between the second position information and the first position information can be obtained in the following manner: obtaining a single-point change value between each boundary pixel point position in the second position information and each boundary pixel point position in the first position information; taking the average value or median of all the single-point change values ​​as the change value between the second position information and the first position information. Because there can be multiple boundary pixels, theoretically the displacement of each boundary pixel point should be the same. If the displacement of each boundary pixel point is different, the final displacement of the object block can be determined by taking the average value or selecting the median.

[0083] Further, in some embodiments of the vehicle abnormal vibration detection method, in step S20, as Figure 2 As shown, the image displacement of two adjacent frames of images of each vehicle-mounted camera can also be obtained in the following manner:

[0084] S20A: Determine the moving direction and moving amount of each pixel point in two adjacent image frames.

[0085] Specifically, the moving direction and amount can be determined by comparing the position changes of each target object in two adjacent frames, or the moving direction and amount of each pixel in two adjacent frames can be determined by using the optical flow method. Optical flow is used to define the movement of objects in an image, which can be caused by camera movement or object movement. Specifically, it refers to the amount of movement of the pixel representing the same object (object) in one frame of a video image to the next frame, represented by a two-dimensional vector. Depending on whether sparse points in the image are selected for optical flow estimation, optical flow estimation can be divided into sparse optical flow and dense optical flow.

[0086] S20B: Group all the pixels, and the pixels in each group have the same moving direction.

[0087] Among them, because the moving directions are divided according to up and down and left and right, the moving directions can be considered to be the same if they are generally the same. For example, upward movement is the vertical direction, and the direction within fifteen degrees to the left and right of the vertical direction can be considered to be upward movement.

[0088] S20C: determining a target pixel group having the most pixels, and fusing the movement amounts of the pixels in the target pixel group to obtain the image displacement amount.

[0089] The fusion algorithm can be implemented by taking the average value or the median. Specifically, the displacement of the pixels in each object block in the image can be calculated by dense optical flow. After obtaining the dense optical flow, the pixels with the largest number of moving directions in the same or similar directions can be counted. Then, the displacement of these pixels can be averaged or the median can be calculated to obtain the overall displacement of the image. Obviously, the overall displacement of the image can represent the displacement of the camera, and the camera is rigidly set on the vehicle, and the displacement of the camera can reflect the degree of vibration of the vehicle.

[0090] In specific implementation, the above step S20A can also be implemented in the following manner:

[0091] SA1: Input two adjacent frames of captured images into a preset displacement recognition model, so that the preset displacement recognition model can analyze the two adjacent frames of captured images using an optical flow method. The preset displacement recognition model is pre-configured with an optical flow method implementation software, and this solution uses a dense optical flow algorithm.

[0092] SA2: Using the output result of the preset displacement recognition model as the object block displacement of the at least one object block. Combined with the above definition of the optical flow method, the preset displacement recognition model can output a two-dimensional vector, that is, recording the displacement of the object block in the row direction and the column direction, so as to determine the object block displacement.

[0093] In some schemes, the preset displacement recognition model can be obtained by: using multiple groups of sample images marked with object blocks as input samples, and using the displacement of the object blocks in each group of sample images as output samples; training a deep learning algorithm based on the input samples and the output samples, and using the trained deep learning algorithm as the preset displacement recognition model. That is, the optical flow method and the deep learning algorithm are combined, so that the recognition result can be obtained more accurately and efficiently.

[0094] In some embodiments, the step S30 of identifying key information of the target object in contact with the vehicle in the abnormal vibration point image includes:

[0095] S301: Identify the vehicle contour line in the abnormal vibration point image.

[0096] The edge pixel points of the part of the vehicle body captured in the abnormal vibration point image constitute the vehicle contour. In addition, because the vehicle camera is installed on the vehicle, the part of the vehicle body captured by it can be determined by factory calibration, that is, in the image captured by the vehicle camera, the pixel points occupied by the vehicle body can be stored as a known quantity. When the captured image is obtained, this part of the pixel points can be divided into the vehicle body, and the vehicle contour is obtained.

[0097] S302: Acquire the contour line of the environmental target in the abnormal vibration point image.

[0098] Environmental targets are pedestrians and vehicles (including various types of vehicles such as cars or tricycles). The contours of the environmental targets can be identified by analyzing the abnormal vibration point image, or the contours of the environmental targets can be obtained by dividing the target blocks as in step S201.

[0099] S303: If the contour line of any of the environmental targets has an intersection with the contour line of the vehicle, the environmental target is used as the target object. It can be understood that the intersection is the location where the collision or scrape occurs.

[0100] Furthermore, in some solutions, the step S30 of identifying key information of the target object in contact with the vehicle in the abnormal vibration point image includes:

[0101] 304: If the target object is a human object, the face information of the human object is identified as the key information, or if the target object is a vehicle outline object, the license plate information of the vehicle outline object is identified as the key information. Through this solution, the key information corresponding to the target object can be acquired and stored for easy subsequent search.

[0102] Furthermore, the detection method in some schemes may also include the following steps:

[0103] S50: Acquire the number of abnormal vibration point images.

[0104] As mentioned above, the abnormal vibration point image refers to an image with a larger displacement compared with the previous frame image, and is counted in units of frames.

[0105] S60: Determine the duration of the abnormal vibration based on the interval time between two adjacent frames of images sent by the vehicle-mounted camera and the number.

[0106] The working cycle of the vehicle-mounted camera is fixed, from which the interval between two adjacent frames of images can be determined, and the duration can be calculated by multiplication.

[0107] S70: If the duration of the abnormal vibration is greater than the vibration duration threshold, it is determined that the vehicle is at risk of being stolen and an alarm prompt signal is issued.

[0108] If it is a collision, it may be impact, and the abnormal vibration lasts for a short time. If it is a theft, the vehicle needs to be moved continuously, and the abnormal vibration lasts for a long time. The duration threshold can be determined according to the vibration time when a collision usually occurs.

[0109] The present application also provides a vehicle abnormal vibration point detection device, such as Figure 4 As shown, including:

[0110] The image acquisition module 10 is configured to acquire continuous multi-frame images sent by multiple vehicle-mounted cameras, and different vehicle-mounted cameras are used to capture different directions of the environment in which the vehicle is located; the present application scheme is applicable to vehicles equipped with surround-view cameras, and the surround-view cameras actually include multiple vehicle-mounted cameras, and different vehicle-mounted cameras are used to capture different directions of the environment in which the vehicle is located. The vehicle-mounted control terminal is connected to the surround-view camera of the vehicle in communication, so the vehicle-mounted control terminal can acquire the images of the environment in which the vehicle is located captured by the surround-view camera in real time.

[0111] The abnormal camera determination module 20 is configured to obtain the image displacement of two adjacent frames of each of the vehicle-mounted cameras. If the image displacement is greater than a set threshold, the corresponding vehicle-mounted camera is determined as an abnormal vibration camera; this solution can determine the displacement of any frame of image compared to the previous frame of image by correlation method, deep learning algorithm, etc. Assuming that the vehicle is equipped with four vehicle-mounted cameras, which respectively capture images in the front, rear, left and right directions of the vehicle, the image displacement of each vehicle-mounted camera can be determined. Under normal circumstances, the vehicle-mounted camera is rigidly set on the vehicle, and if the vehicle is in a parked state, the image captured by the vehicle-mounted camera should also have a relatively constant change or remain unchanged. For example, if the vehicle is parked in a parking lot, there are other parked vehicles or some pedestrians in the parking lot, and the images captured by each camera are either stationary vehicles or stationary or normal pedestrians. Furthermore, if the vehicle is in a driving state, there will be no overall large movement between the continuous images captured instantly by the vehicle-mounted camera, and only some of the pixels therein will move within a normal range.

[0112] The abnormal image determination module 30 is configured to obtain the image sent by the abnormal vibration camera as an abnormal vibration point image, and identify the key information of the target object in contact with the vehicle in the abnormal vibration point image; if the image captured by a certain vehicle-mounted camera moves greatly as a whole, it means that the vehicle-mounted camera moves greatly, and the vehicle-mounted camera is rigidly mounted on the vehicle, and it can be considered that the vehicle has an abnormal vibration. At this time, when the image movement is large, the corresponding vehicle vibration amplitude is large, so the image of the corresponding abnormal vibration camera is selected for the main analysis, and the situation when the abnormal vibration occurs can be recorded. The target object may be a person, a vehicle, or a foreign object such as a stone. When the target object contacts or collides with the vehicle, it will cause the vehicle to vibrate abnormally. When the vehicle vibrates abnormally, the vehicle-mounted camera will vibrate abnormally, resulting in a large displacement of the image captured by the vehicle-mounted camera. The key information will also be different according to the category of the target object. The target object can be determined through the key information. For example, if it is a vehicle, the key information may include the license plate number. When the license plate number is determined, the vehicle can be easily determined.

[0113] The abnormal vibration determination module 40 is configured to obtain the key information of the target object and the integrated information of the corresponding abnormal vibration point image as abnormal vibration detection information, wherein the key information and the corresponding abnormal vibration point image can be processed according to the existing encoding method to obtain a complete data packet as integrated information, or the key information and the corresponding abnormal vibration point image can be stored in the same folder, and the compressed data packet obtained after compressing the folder is used as integrated information, etc. In this solution, when abnormal vibration occurs, the key information of the target object causing the abnormal vibration can be recorded to facilitate the subsequent search for the target object. In some solutions, the key information can also be sent to the target terminal, which can be a server, a user's mobile phone, etc., and the key information can be used as subsequent forensic information.

[0114] The above scheme provided by this embodiment determines the abnormal vibration point of the vehicle and obtains abnormal vibration detection information by analyzing the images taken by the on-board cameras in each direction of the existing surround view cameras. Specifically, the scheme of this application obtains continuous multiple frames of images sent by multiple on-board cameras, analyzes the image displacement of two adjacent frames of images of each on-board camera, and determines it as an abnormal vibration camera if the image displacement of the on-board camera is greater than the set threshold. The image sent by the abnormal vibration camera is used as the abnormal vibration point image, and the key information of the target object in the abnormal vibration point image is extracted. The integrated information of the key information of the target object and the corresponding abnormal vibration point image is obtained as abnormal vibration detection information. The above scheme of this application can determine the on-board camera corresponding to the abnormal vibration point when abnormal vibration is detected in the vehicle, and integrate the image and the target object recorded with the key information of the target object taken by the on-board camera into abnormal vibration detection information as support information for subsequent search for responsible personnel, thereby avoiding losses caused by the inability to clarify the information of the responsible personnel.

[0115] In some schemes, the abnormal camera determination module 20 obtains the set threshold value in the following manner: the average value or median value of the image displacement of all the vehicle-mounted cameras obtained at the same time is used as the set threshold value. In practical applications, for the vehicle-mounted surround view camera, the all-round situation of the front, back, left, and right of the vehicle can be taken into account. The vehicle-mounted cameras set to monitor multiple directions, if the location of the vibration is to be determined, can be determined by comparing the displacement difference of the images captured by each vehicle-mounted camera, that is, the average value or median value of the image displacement of all the vehicle-mounted cameras obtained at the same time can be used as the set threshold value for comparison. For example, if the image displacement of one vehicle-mounted camera is much higher than the image displacement of other vehicle-mounted cameras, then the image displacement of the vehicle-mounted camera must be greater than the set threshold value, indicating that the vibration source is in the shooting direction of this vehicle-mounted camera.

[0116] In some schemes, the abnormal camera determination module 20 obtains the image displacement of two adjacent frames of images of each of the vehicle-mounted cameras in the following manner for each image sent by the vehicle-mounted camera: identifying at least one object block in each frame of the image, the object block including at least one pixel; obtaining the object block displacement of the at least one object block in two adjacent frames of the image; and fusing the object block displacement of the at least one object block to obtain the image displacement of two adjacent frames of the image. The object block in the image is a position point or area including at least one pixel. For example, other vehicles, trees, etc. in the environment. The image can be analyzed by recognition algorithms such as artificial intelligence to analyze the various object blocks contained in the image. The position of the object block in the image can be determined by the position of the pixels it occupies. For example, the pixels included in the object block are in the third to fifth rows and the seventh to ninth columns in the image, thereby determining the position of the object block in the image. Because the intervals between pixels in different rows and different columns of the image can be determined by the resolution of the vehicle-mounted camera. Therefore, after comparing two adjacent frames of images, we can determine how many rows and columns the object block has moved in the row direction and how many columns it has moved in the column direction. Combining the known row and column intervals, we can get the displacement. In order to simplify the calculation, we can also directly use the number of rows and columns moved as the displacement. The fusion method can be set. For example, if there are multiple object blocks in the image, the average displacement of each object block can be used to calculate the overall displacement of the image.

[0117] Under normal circumstances, if the image displacement between two adjacent frames of images taken by the vehicle camera is too large, it can be determined that the vehicle has experienced abnormal vibration, which may be caused by theft or collision. This solution can determine the image displacement by measuring the object block displacement of the object blocks in the continuous multiple frames of images taken by the vehicle camera. Since the vehicle camera is a component configured by the vehicle itself, the line connection between it and the various control modules on the vehicle has been completed, and there is no need to redesign and arrange the lines.

[0118] Specifically, the abnormal camera determination module 20 is used to obtain the pixel value of each pixel point at each position in each frame of the image, obtain the difference between the pixel value of any current position pixel point and the pixel value of the adjacent position pixel point, if the difference is less than the set pixel value, the current position pixel point and the adjacent position pixel point are classified as the same object block, otherwise the current position pixel point is used as the boundary pixel point of the object block where the current position pixel point is located. The pixel value is used to represent the value assigned by the processor when the original image is digitized, which can represent the average brightness information of the small square of pixel points in the original image, or the average reflection / transmission density information of the small square, etc. The relevant parts involving pixel values ​​in this application can use the same parameters, for example, brightness information is used to represent pixel values. The difference can be obtained by directly subtracting the pixel values ​​of two pixels. If the difference between the pixel values ​​of two adjacent pixels is not large, they can be considered to be the same object block, otherwise they are considered to be not the same object block. Specifically in actual application scenarios, it can be considered that the pixel values ​​of a certain object block should have a certain balance, for example, when the road surface in the image is used as an object block, the pixel values ​​should be approximately equal. For image processing, obtaining the pixel value of each pixel in the image can be achieved by using a relatively simple algorithm. This method can simplify the calculation and improve the detection efficiency.

[0119] Further, the abnormal camera determination module 20 is used to obtain the first position information of each boundary pixel point of any object block in the previous frame image in the previous frame image. The second position information of each boundary pixel point of the same object block in the current frame image is obtained. The change value between the second position information and the first position information is obtained as the displacement of the object block. After identifying each object block in the previous frame image, the first position information of each boundary pixel point of any object block in the previous frame image can be determined. It can be expressed in the form of the number of rows and columns where the pixel point is located. After identifying each object block in the current frame image, the second position information of each boundary pixel point of any object block in the current frame image can be determined. It can be expressed in the form of the number of rows and columns where the pixel point is located. As mentioned above, because the intervals between different row pixels and different column pixels of the image can be determined by the resolution of the vehicle-mounted camera. Therefore, after comparing two adjacent frames of images, it can be determined how many rows the object block has moved in the row direction and how many columns it has moved in the column direction. Combined with the known row interval and column interval, the displacement can be obtained. In order to simplify the calculation, the number of rows and columns moved can also be directly used as the displacement. Specifically, the change value between the second position information and the first position information can be obtained in the following way: obtain the single-point change value between each boundary pixel point position in the second position information and each boundary pixel point position in the first position information; take the average or median of all the single-point change values ​​as the change value between the second position information and the first position information. Because there can be multiple boundary pixels, theoretically the displacement of each boundary pixel should be the same. If the displacement of each boundary pixel is different, the final object block displacement can be determined by taking the average value or selecting the median.

[0120] Further, the abnormal vibration detection method of the vehicle described in some schemes, the abnormal camera determination module 20, is also used to obtain the image displacement of two adjacent frames of images of each of the vehicle-mounted cameras in the following manner: determine the moving direction and movement amount of each pixel point in the two adjacent frames of images; group all the pixels, and the pixels in each group have the same moving direction; determine the target pixel group with the most pixels, and merge the movement amounts of each pixel point in the target pixel group to obtain the image displacement amount. Specifically, the moving direction and movement amount can be determined by comparing the position changes of each target object in the two adjacent frames of images, or the moving direction and movement amount of each pixel point in the two adjacent frames of images can be determined by using the optical flow method. Optical flow is used to define the movement of objects in the image, which can be caused by camera movement or object movement. Specifically, it refers to the movement amount of the pixel point representing the same object (object) in one frame of the video image moving to the next frame, which is represented by a two-dimensional vector. Depending on whether sparse points in the image are selected for optical flow estimation, optical flow estimation can be divided into sparse optical flow and dense optical flow. Because the moving directions are divided according to up and down and left and right, the moving directions can be considered to be the same if they are generally the same. For example, if the vertical direction is upward, the direction within fifteen degrees of the vertical direction can be considered to be upward. The fusion algorithm can be implemented by taking the average value or the median. Specifically, the displacement of the pixels in each object block in the image can be calculated by dense optical flow. After obtaining the dense optical flow, the largest number of pixels with the same or similar moving directions can be counted. Then, the displacement of these pixels is averaged or the median is calculated to obtain the overall displacement of the image. Obviously, the overall displacement of the image can represent the displacement of the camera, and the camera is rigidly set on the vehicle, and the displacement of the camera can reflect the degree of vibration of the vehicle.

[0121] In specific implementation, the abnormal camera determination module 20 is also used to input two adjacent frames of captured images into a preset displacement recognition model, so that the preset displacement recognition model can analyze the two adjacent frames of captured images using the optical flow method. Among them, the preset displacement recognition model is pre-configured with the implementation software of the optical flow method, and this scheme adopts the dense optical flow algorithm; the output result of the preset displacement recognition model is used as the object block displacement of the at least one object block. Combined with the above definition of the optical flow method, the preset displacement recognition model can output a two-dimensional vector, that is, it records the displacement of the object block in the row direction and the column direction, so as to determine the displacement of the object block. In some schemes, the above preset displacement recognition model can be obtained by the following method: using multiple groups of sample images marked with object blocks as input samples, and using the object block displacement in each group of sample images as output samples; training the deep learning algorithm based on the input samples and the output samples, and using the trained deep learning algorithm as the preset displacement recognition model. That is, the optical flow method and the deep learning algorithm are combined, so that the recognition result can be obtained more accurately and efficiently.

[0122] In some schemes, the abnormal image determination module 30 is used to identify the contour line of the vehicle in the abnormal vibration point image; obtain the contour line of the environmental target in the abnormal vibration point image; if the contour line of any of the environmental targets has an intersection with the contour line of the vehicle, the environmental target is used as the target object; further, the abnormal image determination module 30 is also used to: if the target object is a human object, identify the face information of the human object as the key information, or, if the target object is a vehicle contour object, identify the license plate information of the vehicle contour object as the key information. The part of the body of the vehicle captured in the abnormal vibration point image and the edge pixel points of the part of the body constitute the contour line of the vehicle. In addition, because the vehicle-mounted camera is installed on the vehicle, the part of the body of the vehicle captured by it can be determined by the factory calibration method, that is, in the image captured by the vehicle-mounted camera, the pixel points occupied by the body of the vehicle can be stored as a known quantity. When the captured image is obtained, this part of the pixel points can be divided into the body of the vehicle, and the contour line of the vehicle is obtained. Environmental targets are pedestrians and vehicles (including various types of vehicles such as cars or tricycles). The contours of the environmental targets can be identified by analyzing the abnormal vibration point image, or the contours of the environmental targets can be obtained by dividing the target blocks as described above. The intersection is the location where the collision or scratch occurs. Through this solution, the key information corresponding to the target object can be obtained and stored for subsequent search.

[0123] In some embodiments, the device further comprises:

[0124] The type judgment module is configured to obtain the number of abnormal vibration point images, determine the duration of abnormal vibration based on the interval time between two adjacent frames of images sent by the vehicle-mounted camera and the number, and if the duration of abnormal vibration is greater than the vibration duration threshold, it is determined that the vehicle is at risk of being stolen and an alarm prompt signal is issued. As mentioned above, the abnormal vibration point image refers to an image with a larger displacement compared to the previous frame image, which is counted in frames. The working cycle of the vehicle-mounted camera is determined, so the interval between two adjacent frames of images can be determined, and the duration can be calculated by multiplication. If it is a collision category, it may be impact-like, and the duration of abnormal vibration is short. If it is theft, the vehicle needs to be continuously moved, and the duration of abnormal vibration is long. The duration threshold can be determined based on the vibration time when a collision usually occurs.

[0125] An embodiment of the present application also provides a computer-readable storage medium, in which program information is stored. After a computer reads the program information, the vehicle abnormal vibration detection method provided in the above method embodiment is executed.

[0126] The present application also provides an electronic device, such as Figure 5As shown, the electronic device includes at least one processor 51 and at least one memory 52, at least one of the memories 52 stores program information, and at least one of the processors 51 reads the program information and executes the vehicle abnormal vibration detection method described in any one of the above method embodiments. The device may also include: an input device 53 and an output device 54. The processor 51, the memory 52, the input device 53 and the output device 54 can be connected in communication. The memory 52, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 51 executes various functional applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 52, that is, the vehicle abnormal vibration detection method provided by any of the above schemes is implemented. The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the vehicle abnormal vibration detection method, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to a device for executing the vehicle abnormal vibration detection method through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof. The input device 53 may receive input user clicks, and generate signal inputs related to user settings and function controls of the vehicle abnormal vibration detection method. The output device 54 may include a display device such as a display screen. When the one or more modules are stored in the memory 52, when they are run by the one or more processors 51, the vehicle abnormal vibration detection method in any of the above-mentioned method embodiments is executed.

[0127] An embodiment of the present application further provides a vehicle, wherein the vehicle is equipped with the vehicle vibration detection device or the computer-readable storage medium or the electronic device described in any one of the aforementioned embodiments.

[0128] As needed, the above technical solutions can be combined to achieve the best technical effect.

[0129] The above are only the principles and preferred embodiments of the present application. It should be noted that, for ordinary technicians in this field, on the basis of the principles of the present application, several other modifications can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for detecting abnormal vibration of a vehicle, characterized in that: include: Acquire continuous multi-frame images sent by multiple vehicle-mounted cameras, where different vehicle-mounted cameras are used to capture different directions of the environment where the vehicle is located; Obtaining the image displacement of two adjacent frames of images of each of the vehicle-mounted cameras, and if the image displacement is greater than a set threshold, determining the corresponding vehicle-mounted camera as an abnormal vibration camera; Acquire the image sent by the abnormal vibration camera as an abnormal vibration point image, and identify key information of the target object in contact with the vehicle in the abnormal vibration point image; The integrated information of the key information of the target object and the corresponding abnormal vibration point image is obtained as abnormal vibration detection information.

2. The vehicle abnormal vibration detection method according to claim 1, characterized in that: The image displacement of two adjacent frames of images of each vehicle-mounted camera is obtained, and if the image displacement is greater than a set threshold, the corresponding vehicle-mounted camera is determined to be an abnormal vibration camera, and the set threshold is obtained in the following manner: The average value or median value of the image displacements of all the vehicle-mounted cameras obtained at the same time is used as the set threshold.

3. The vehicle abnormal vibration detection method according to claim 1, characterized in that: The key information for identifying the target object in contact with the vehicle in the abnormal vibration point image includes: Identifying the contour line of the vehicle in the abnormal vibration point image; Acquiring the contour line of the environmental target in the abnormal vibration point image; If the contour line of any of the environmental objects has an intersection with the contour line of the own vehicle, the environmental object is taken as the target object.

4. The vehicle abnormal vibration detection method according to claim 1, characterized in that: The key information for identifying the target object in contact with the vehicle in the abnormal vibration point image includes: If the target object is a human object, the facial information of the human object is identified as the key information; or, if the target object is a vehicle outline object, the license plate information of the vehicle outline object is identified as the key information.

5. The vehicle abnormal vibration detection method according to claim 1, characterized in that: The step of obtaining the image displacement of two adjacent frames of images of each of the vehicle-mounted cameras includes: Determine the moving direction and moving amount of each pixel in two adjacent frames of images; All pixels are grouped, and the pixels in each group have the same moving direction; A target pixel group having the most pixels is determined, and the movement amounts of the pixels in the target pixel group are merged to obtain the image displacement amount.

6. The vehicle abnormal vibration detection method according to any one of claims 1 to 5, characterized in that: After associating the key information of the target object with the abnormal vibration point image as abnormal vibration detection information, the method further includes: Acquiring the number of abnormal vibration point images; Determine the duration of abnormal vibration based on the interval time between two adjacent frames of images sent by the vehicle-mounted camera and the number of abnormal vibration point images; If the abnormal vibration lasts longer than the vibration duration threshold, it is determined that the vehicle is at risk of being stolen and an alarm signal is issued.

7. A vehicle abnormal vibration detection device, characterized in that: include: An image acquisition module is configured to acquire continuous multi-frame images sent by a plurality of vehicle-mounted cameras, wherein different vehicle-mounted cameras are used to capture images in different directions of the environment where the vehicle is located; an abnormal camera determination module, configured to obtain an image displacement of two adjacent frames of images of each of the vehicle-mounted cameras, and if the image displacement is greater than a set threshold, determine the corresponding vehicle-mounted camera as an abnormal vibration camera; an abnormal image determination module, configured to obtain the image sent by the abnormal vibration camera as an abnormal vibration point image, and identify key information of a target object in contact with the vehicle in the abnormal vibration point image; The abnormal vibration determination module is configured to obtain integrated information of the key information of the target object and the corresponding abnormal vibration point image as abnormal vibration detection information.

8. A computer-readable storage medium, characterized in that: The storage medium stores program information, and the computer reads the program information and executes the vehicle abnormal vibration detection method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device includes at least one processor and at least one memory, at least one of the memories stores program information, and at least one of the processors reads the program information and executes the vehicle abnormal vibration detection method according to any one of claims 1-6.

10. A vehicle, characterized in that: The vehicle is equipped with the vehicle abnormal vibration detection device according to claim 7, the computer-readable storage medium according to claim 8, or the electronic device according to claim 9.

Images