Vehicle parking monitoring method, device and system and vehicle

Abstract

The invention relates to a parking monitoring method, device and system for a vehicle and the vehicle, and the method comprises the steps: responding to the vehicle entering a parking monitoring mode, controlling an image processing unit of a vehicle-mounted system on chip (SOC) to sleep, and indicating a vehicle-mounted camera to enter a first monitoring mode; the vehicle-mounted camera acquires a reference image frame and a to-be-detected image frame in a first monitoring mode, performs image difference comparison on the reference image frame and the to-be-detected image frame, and sends an image frame set to the controller under the condition that the image difference meets a first condition, and the image frame set comprises the reference image frame and the to-be-detected image frame; and in response to the received image frame set, waking up the image processing unit to enable the image processing unit to perform exception monitoring based on the image frame set. Compared with a vehicle-mounted SOC image processing unit which is always kept in an awakening state, energy consumption can be remarkably reduced, and the endurance mileage of the vehicle can be increased through reduction of the energy consumption.

Description

Technical Field

[0001] This application relates to the field of vehicle technologies, in particular to the field of parking monitoring technologies, and specifically to a parking monitoring method, device, system and vehicle for a vehicle. Background Art

[0002] With the development of automotive technologies, in order to solve the parking safety problem of vehicles, the sentry mode of vehicles has been widely applied in vehicles. The sentry mode means that when the vehicle is in the parked state, the cameras on the vehicle record the surrounding environment of the vehicle in real time. Once it is recognized that the vehicle has been collided, damaged, etc., the recorded video information is notified to the vehicle owner at the same time. However, in the sentry mode, usually the power consumption of the vehicle is relatively high, and the requirement for storage space is high. Summary of the Invention

[0003] This application provides a parking monitoring method, device, system and vehicle for a vehicle, so as to significantly reduce energy consumption on the premise of ensuring the effectiveness of monitoring during the parking monitoring process.

[0004] According to the first aspect provided by this application, there is provided a parking monitoring method for a vehicle, which is applied to a controller of the vehicle. The method includes: In response to the vehicle entering the parking monitoring mode, control the image processing unit of the system on chip (SOC) in the vehicle to go to sleep, and instruct the on-vehicle camera to enter the first monitoring mode; in the first monitoring mode, the on-vehicle camera acquires a reference image frame and a to-be-tested image frame, performs image difference comparison on the reference image frame and the to-be-tested image frame, and in the case where the image difference meets the first condition, sends an image frame set to the controller, and the image frame set includes: the reference image frame and the to-be-tested image frame; In response to receiving the image frame set, wake up the image processing unit so that the image processing unit performs anomaly monitoring based on the image frame set.

[0005] Adopting the solution of the embodiment of this application, after the vehicle enters the parking monitoring mode, control the image processing unit of the on-vehicle SOC to go to sleep, and instruct the on-vehicle camera to enter the first monitoring mode. The on-vehicle camera performs difference comparison on the acquired image frames, and only sends the image frames to the controller when it is initially confirmed that the image difference is large. The controller then wakes up the image processing unit, and the image processing unit further analyzes the image frames.

[0006] Thus, the image frames are only sent to the controller when in-depth analysis of the image frames is required, significantly reducing the data transmission volume between the on-vehicle camera and the controller, and reducing the energy consumption caused by data transmission.

[0007] In addition, in the first monitoring mode, the image processing unit of the in-vehicle SOC is in a sleep state most of the time, and is only awakened by the controller when the in-vehicle camera determines that the difference between image frames is large. Compared with the image processing unit of the in-vehicle SOC always being in an awakened state, it can significantly reduce energy consumption. The reduction in energy consumption can increase the driving range of the vehicle.

[0008] In a possible way, the image processing unit includes a first processing subunit and a second processing subunit; Waking up the image processing unit in response to receiving a set of image frames includes: Waking up the first processing subunit in response to receiving a set of image frames, so that the first processing subunit performs anomaly determination based on the set of image frames; The method further includes: In the case where the determination result of the anomaly determination shows an anomaly, or the confidence level of the determination result of the anomaly determination is less than a preset threshold, instruct the in-vehicle camera to enter the second monitoring mode; in the second monitoring mode, the in-vehicle camera acquires a monitoring video stream and sends the monitoring video stream to the controller; Waking up the second processing subunit in response to receiving the monitoring video stream, so that the second processing subunit performs anomaly monitoring based on the monitoring video stream.

[0009] Adopting the solution of the embodiment of the present application, the controller first wakes up the first processing subunit of the in-vehicle SOC. In the case where the determination result made by the first processing subunit shows an anomaly, or the first processing subunit cannot give a determination result with a high confidence level, instruct the in-vehicle camera to enter the second monitoring mode. The in-vehicle camera acquires a monitoring video stream at a higher resolution and frame rate and sends it to the controller. The controller wakes up the second processing subunit, and the second processing subunit performs a more in-depth anomaly determination based on its stronger image processing ability compared with the first processing subunit to obtain an anomaly determination result with a high confidence level.

[0010] It can be seen that it is possible to achieve anomaly monitoring during parking monitoring, obtain an anomaly determination result with a high confidence level, and reduce the probability of missing abnormal situations while saving power as much as possible.

[0011] In a possible way, the image processing ability and energy consumption of the first processing subunit are respectively less than those of the second processing subunit.

[0012] In a possible way, the frame rate and image resolution of the images collected by the in-vehicle camera in the first monitoring mode are respectively less than the frame rate and image resolution of the images collected by the in-vehicle camera in the second monitoring mode.

[0013] In a possible way, the image difference satisfying the first condition indicates that the evaluation value of the image difference between the reference image frame and the image frame to be measured exceeds a preset evaluation threshold.

[0014] In a possible way, the vehicle-mounted camera compares the reference image frame and the image frame to be measured for image difference, including: the vehicle-mounted camera compares the pixels of the reference image frame and the image frame to be measured to obtain difference pixels.

[0015] In a possible way, the image difference satisfying the first condition indicates that the proportion of the difference pixels between the reference image frame and the image frame to be measured exceeds a preset threshold.

[0016] In a possible way, the image frame set includes: the image frames of the first region of the reference image frame and the image frames of the first region of the image frame to be measured; wherein, the first region is the image region where the difference pixels between the reference image frame and the image frame to be measured are located.

[0017] According to the second aspect provided by the present application, a parking monitoring device for a vehicle is provided. The device includes: A control module, configured to, in response to the vehicle entering the parking monitoring mode, control the image processing unit of the vehicle-mounted SOC to enter the sleep state, and instruct the vehicle-mounted camera to enter the first monitoring mode; in the first monitoring mode, the vehicle-mounted camera acquires a reference image frame and an image frame to be measured, compares the reference image frame and the image frame to be measured for image difference, and when the image difference satisfies the first condition, sends an image frame set to the vehicle-mounted SOC, and the image frame set includes: the reference image frame and the image frame to be measured; A wake-up module, configured to, in response to receiving the image frame set, wake up the image processing unit so that the image processing unit performs abnormal monitoring based on the image frame set.

[0018] In a possible way, the image processing unit includes a first processing subunit and a second processing subunit; The wake-up module is specifically configured to: In response to receiving the image frame set, wake up the first processing subunit so that the first processing subunit performs abnormal determination based on the image frame set; The device further includes an indication module, configured to: In the case where the determination result of the abnormal determination shows an abnormality, or the confidence level of the determination result of the abnormal determination is less than a preset threshold, instruct the vehicle-mounted camera to enter the second monitoring mode; in the second monitoring mode, the vehicle-mounted camera acquires a monitoring video stream and sends the monitoring video stream to the vehicle-mounted SOC; The wake-up module is further configured to: In response to receiving the monitoring video stream, wake up the second processing subunit so that the second processing subunit performs abnormal monitoring based on the monitoring video stream.

[0019] In one possible way, the image processing capabilities and energy consumption of the first processing subunit are respectively less than those of the second processing subunit.

[0020] In one possible way, the frame rate and image resolution of the vehicle-mounted camera for collecting images in the first monitoring mode are respectively less than those of the vehicle-mounted camera for collecting images in the second monitoring mode.

[0021] In one possible way, the image difference satisfying the first condition indicates that the image difference evaluation value between the reference image frame and the image frame to be measured exceeds a preset evaluation threshold.

[0022] In one possible way, the image difference satisfying the first condition indicates that the proportion of the difference pixels between the reference image frame and the image frame to be measured exceeds a preset threshold.

[0023] In one possible way, the image frame set includes: the image frames of the first region of the reference image frame and the image frames of the first region of the image frame to be measured; wherein, the first region is the image region where the difference pixels between the reference image frame and the image frame to be measured are located.

[0024] According to the third aspect provided by the present application, a parking monitoring system for a vehicle is provided. The parking monitoring system includes: a controller, a vehicle-mounted SOC, and a vehicle-mounted camera; The controller is configured to, in response to the vehicle entering the parking monitoring mode, control the image processing unit of the vehicle-mounted SOC to enter the sleep state, and instruct the vehicle-mounted camera to enter the first monitoring mode; The vehicle-mounted camera is configured to, in the first monitoring mode, obtain a reference image frame and an image frame to be measured, perform an image difference comparison on the reference image frame and the image frame to be measured, and when the image difference satisfies the first condition, send an image frame set to the controller. The image frame set includes: the reference image frame and the image frame to be measured; The controller is further configured to, in response to receiving the image frame set, wake up the image processing unit so that the image processing unit performs abnormal monitoring based on the image frame set.

[0025] According to the fourth aspect provided by the present application, a vehicle is provided, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the instructions to implement the method of the first aspect and any possible implementation manner thereof.

[0026] According to the fifth aspect provided by the present application, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the method of the first aspect and any possible implementation manner thereof.

[0027] According to a sixth aspect provided by the present application, there is provided a computer program product, which includes computer instructions. When the computer instructions run on an electronic device, the electronic device is caused to execute the method according to the first aspect and any possible implementation manner thereof as described above.

[0028] It should be noted that for the technical effects brought by any implementation manner in the second aspect to the sixth aspect, reference may be made to the technical effects brought by the corresponding implementation manner in the first aspect, which will not be elaborated here.

[0029] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application, and do not constitute an improper limitation to the present application.

[0031] Figure 1 is a schematic flowchart of a parking monitoring method for a vehicle shown according to an exemplary embodiment; Figure 2 is a schematic diagram of the connection relationship between a camera module and a controller shown according to an exemplary embodiment; Figure 3 is a schematic structural diagram of an in-vehicle camera shown according to an exemplary embodiment; Figure 4 is a schematic flowchart of another parking monitoring method for a vehicle shown according to an exemplary embodiment; Figure 5 is a schematic structural diagram of a parking monitoring system for a vehicle shown according to an exemplary embodiment; Figure 6 is a block diagram of a parking monitoring device for a vehicle shown according to an exemplary embodiment; Figure 7 is a block diagram of an electronic device shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0033] It should be noted that the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of this application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] In the embodiments of this application, words such as "exemplary", "such as", or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary", "such as", or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary", "such as", or "for example" is intended to present relevant concepts in a specific manner.

[0035] In the current implementation solutions of the sentry mode, it is generally manually enabled by the user. After the sentry mode is enabled, the SOC of the vehicle is in a working state, and the processor power consumption is huge, resulting in large power consumption; and after the sentry mode is enabled, the camera is also in a working state, which also consumes power.

[0036] There are generally two common ways to reduce power consumption. The first is to combine with other sensors of the vehicle (such as radar, vibration sensor) to achieve. The camera is only turned on when the radar or vibration sensor detects an abnormality. However, in this implementation, the radar or vibration sensor itself also consumes some power, and there is generally a certain delay when the camera is turned on after the radar or vibration sensor detects an abnormality. Assuming that the SOC is in the STR (suspend to RAM) sleep mode, it generally takes 5s to start temporarily, and it is easy to miss sudden events. The second is to judge the cameras that need to be turned on according to the actual situation after the sentry mode is triggered, but essentially the SOC is still in a high power consumption state, the reduced power consumption is limited, and the storage space requirement is still very large.

[0037] To solve the above technical problems, this application provides a parking monitoring method, device, system and vehicle for a vehicle.

[0038] Next, the technical solutions in the embodiments of this application will be described with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments.

[0039] In the embodiments of the present application, a vehicle may also be referred to as a means of transportation (vehicle), a mobile carrier, an electric vehicle (EV), a hybrid electric vehicle (HEV), a plug-in hybrid electric vehicle (PHEV), a fuel cell vehicle (FCV), an autonomous vehicle, an intelligent and connected vehicle (ICV), a driverless vehicle, etc.

[0040] In the embodiments of the present application, the vehicle may be a sedan, a sport utility vehicle (SUV), a truck, an electric vehicle, a motorcycle, a tricycle, a special vehicle (such as an ambulance, a fire truck, a police car, etc.), a driverless taxi, an intelligent and connected bus, an autonomous logistics vehicle, an electric truck, etc. In addition, the method is also applicable to various special vehicles, such as agricultural vehicles, mining vehicles, forestry vehicles, airport vehicles, port vehicles, etc. The present application does not make specific limitations in this regard.

[0041] For the sake of easy understanding, the following specifically introduces the vehicle parking monitoring method provided by the present application with reference to the accompanying drawings. Refer to Figure 1 , the method may include the following steps: S101: In response to the vehicle entering the parking monitoring mode, control the image processing unit of the in-vehicle SOC to sleep, and instruct the in-vehicle camera to enter the first monitoring mode; in the first monitoring mode, the in-vehicle camera acquires a reference image frame and a to-be-tested image frame, compares the reference image frame and the to-be-tested image frame for image differences, and when the image differences meet the first condition, sends an image frame set to the in-vehicle SOC, and the image frame set includes: the reference image frame and the to-be-tested image frame.

[0042] The in-vehicle SOC is one of the core technologies in the field of vehicle electronics. By integrating functional modules such as a processor, a memory, a communication interface, and a dedicated accelerator onto a single chip, it provides high-performance computing capabilities for intelligent vehicles.

[0043] The parking monitoring method for a vehicle provided by an embodiment of this application can be applied to a vehicle, specifically to a controller of the vehicle. The controller can be an electronic control unit (ECU) of the vehicle. It can be understood that currently, domain controllers are mostly used in vehicles for overall vehicle control. The controller mentioned in the embodiments of this application can also be understood as a domain controller.

[0044] In the embodiments of this application, the controller can be integrated on the in-vehicle SOC or can be a control unit independent of the in-vehicle SOC. The embodiments of this application do not make any limitations in this regard.

[0045] The in-vehicle SOC is integrated with an image processing unit, specifically it can be a graphics processing unit (GPU), an Image Signal Processor (ISP), etc., which are used to analyze and process image data.

[0046] For ease of understanding, the transmission of video stream data is described in conjunction with Figure 2 As shown in the figure, the in-vehicle camera module sends the collected image data to the domain controller through a high-speed serial bus, for example, sending serial data through the gigabit multimedia serial links (GMSL) or the Flat Panel Display Link (FPD-link) protocol. After receiving the serial image data, the domain controller performs format conversion on the serial data through a deserialization chip and then transmits it to the SOC. Figure 2 In the embodiments of this application, the controller of the vehicle can automatically control the vehicle to enter the parking monitoring mode according to the vehicle's working conditions, such as the engine state, the motor state, the parking position, etc. Or, the controller of the vehicle responds to a control instruction sent by the user and controls the vehicle to enter the parking monitoring mode.

[0047] In the parking monitoring mode, after the vehicle is turned off and locked, the system still keeps some functions running to monitor the state of the vehicle's surrounding environment. The main purpose is to provide additional safety protection when the vehicle owner leaves.

[0048] Exemplarily, the parking monitoring mode can include the Sentry Mode.

[0049] In the embodiments of this application, after the controller of the vehicle determines that the vehicle enters the parking monitoring mode, it controls the image processing unit of the in-vehicle SOC to go to sleep and instructs the in-vehicle camera to enter the first monitoring mode.

[0050]

[0051] ​It is understandable that in the parking monitoring mode, most of the functional units of the vehicle SOC will be in a dormant state. However, in the relevant technology, after the vehicle enters the parking monitoring mode, the vehicle camera will continue to collect image data of the vehicle's surrounding environment and send the image data to the controller, so that the image processing unit of the vehicle SOC needs to continuously process the received image data, analyze the image data, and determine whether there are abnormal situations such as collision, damage, etc.

[0052] In the embodiment of the present application, after the vehicle enters the parking monitoring mode, the image processing unit of the vehicle-mounted SOC is controlled to sleep, and the image processing unit of the vehicle-mounted SOC is awakened only when necessary. In addition, the vehicle-mounted camera is controlled to enter the first monitoring mode.

[0053] In the embodiment of the present application, the vehicle-mounted camera continuously collects image data of the vehicle's surrounding environment in the first monitoring mode. Specifically, the image frame collected for the first time after entering the first monitoring mode can be used as a reference image frame, and the image frame collected after the reference image frame can be used as a test image frame.

[0054] It is understandable that multiple vehicle-mounted cameras can be arranged in a vehicle. In the embodiment of the present application, only one of the vehicle-mounted cameras is used as an example for explanation, and the data interaction between other vehicle-mounted cameras and the controller and the vehicle-mounted SOC is similar.

[0055] For ease of understanding, combined Figure 3 A brief description of the vehicle camera module is given below. Figure 3 As shown, The vehicle camera module can be divided into three parts according to the internal logic function, namely, the camera sensor, the serializer chip and the storage chip, such as FLASH or electrically erasable programmable read only memory (EEPROM). Among them, the camera sensor and the serializer chip can communicate based on the mobile industry processor interface (MIPI) protocol, and the camera sensor and the storage chip can communicate based on the serial peripheral interface (SPI) protocol.

[0056] Among them, the ISP chip can be integrated in the sensor, or the ISP can be not integrated in the sensor, and an independent ISP chip is set in the camera module. Generally speaking, the ISP chip has data processing capabilities such as noise reduction, interpolation, bad pixel detection, motion detection, etc.

[0057] In the embodiments of the present application, when the image difference satisfies the first condition, an image frame set is sent to the controller.

[0058] The fact that the image difference satisfies the first condition indicates that the image difference evaluation value between the reference image frame and the image frame to be measured exceeds a preset evaluation threshold.

[0059] It can be understood that in the parking monitoring mode, if there are no other passing vehicles, pedestrians, etc. around the vehicle, there will be no significant differences between the image frames captured by the in-vehicle camera, and there will be no abnormal situations such as collisions or damages at this time.

[0060] Therefore, in the embodiments of the present application, when the image difference is small, there is no need to send image data to the controller, and thus the image processing unit of the in-vehicle SOC will not be awakened. Only when the image difference is large, image data is sent to the controller.

[0061] In the embodiments of the present application, the in-vehicle camera can use its own integrated ISP chip to compare the image difference between the reference image frame and the image frame to be measured to obtain an image difference evaluation value.

[0062] Exemplarily, the method for evaluating the image difference can be, but is not limited to, pixel-based difference comparison, feature-based difference comparison, etc. Among them, pixel-based difference comparison directly compares the pixel values at corresponding positions of the images. Feature-based difference comparison extracts key features in the images for comparison. The embodiments of the present application do not limit this.

[0063] In some embodiments of the present application, a static random-access memory (SRAM) cache is integrated in the ISP chip of the in-vehicle camera. Since the reference image frame is used every time during the process of comparing the image difference, the reference image frame can be stored in the SRAM cache to improve the efficiency of comparing the image difference.

[0064] When it is determined that the image difference evaluation value exceeds the preset evaluation threshold, the in-vehicle camera can package the reference image frame and the image frame to be measured and send them to the controller.

[0065] S102: In response to receiving the image frame set, wake up the image processing unit so that the image processing unit performs abnormal monitoring based on the image frame set.

[0066] As described above, the controller receives serial image frame data through a serial link, performs format conversion on the image frame data, and then transmits it to the in-vehicle SOC.

[0067] In the embodiments of the present application, when the controller receives the data of the image frame set, in addition to performing the above-mentioned format conversion operation, it also wakes up the image processing unit of the in-vehicle SOC.

[0068] The in-vehicle SOC receives the data of the set of image frames after format conversion, and the image processing unit in the in-vehicle SOC further analyzes and processes the data of the set of image frames for anomaly monitoring.

[0069] It can be understood that the image processing capability of the image processing unit in the in-vehicle SOC is much greater than that of the in-vehicle camera. Therefore, after the image processing unit in the in-vehicle SOC is awakened, the image processing unit can conduct a more in-depth analysis on the reference image frame and the image frame to be measured to determine whether an abnormal situation occurs.

[0070] Adopting the solution of the embodiment of the present application, after the vehicle enters the parking monitoring mode, the image processing unit of the in-vehicle SOC is controlled to sleep, and the in-vehicle camera is instructed to enter the first monitoring mode. The in-vehicle camera performs differential comparison on the collected image frames and sends the image frames to the controller only when it is initially confirmed that the image differences are large. The controller then wakes up the image processing unit, and the image processing unit further analyzes the image frames.

[0071] Thus, the image frames are sent to the controller only when in-depth analysis of the image frames is required, greatly reducing the data transmission volume between the in-vehicle camera and the controller and reducing the energy consumption generated by data transmission.

[0072] In addition, in the first monitoring mode, the image processing unit of the in-vehicle SOC is in a sleep state most of the time, and is only awakened by the controller when the in-vehicle camera determines that the differences between the image frames are large. Compared with the image processing unit of the in-vehicle SOC always remaining in the awakened state, the energy consumption can be significantly reduced. The reduction of energy consumption can improve the driving range of the vehicle.

[0073] In some embodiments of the present application, the image processing unit includes a first processing subunit and a second processing subunit. When the controller receives the set of image frames, it first wakes up the first processing subunit, and the first processing subunit first conducts anomaly determination based on the set of image frames. If the first processing subunit can determine that there is no anomaly, there is no need to wake up the second processing subunit.

[0074] Specifically, the image processing capability of the first processing subunit is less than that of the second processing subunit. Correspondingly, the energy consumption generated when the first processing subunit is running is also less than that generated when the second processing subunit is running.

[0075] Exemplarily, the first processing subunit may be an ISP chip, and the second processing subunit may be a GPU chip. Or the first processing subunit and the second processing subunit may be two relatively independent image processing units of an ISP chip. The embodiments of the present application do not make any limitations in this regard.

[0076] In the embodiments of the present application, the first processing subunit determines anomalies based on a set of image frames. The determination results may include: no anomaly, anomaly occurred, and unable to determine whether an anomaly occurred. Specifically, when making a determination result, the first processing subunit generates a confidence level for the determination result. When the confidence level is greater than a specific value, the determination result is considered accurate. If the confidence level of the determination result of the anomaly determination is less than a preset threshold, it is regarded that the first processing subunit is unable to determine whether an anomaly occurred.

[0077] In the embodiments of the present application, when the determination result of the anomaly determination is that an anomaly occurred, or when the confidence level of the determination result of the anomaly determination is less than the preset threshold, the in-vehicle camera is instructed to enter the second monitoring mode. In the second monitoring mode, the in-vehicle camera acquires a monitoring video stream and sends the monitoring video stream to the controller.

[0078] In some embodiments of the present application, the frame rate and image resolution of the in-vehicle camera for collecting images in the first monitoring mode are respectively lower than those of the in-vehicle camera for collecting images in the second monitoring mode.

[0079] Specifically, considering that after the vehicle enters the parking monitoring mode, abnormal events such as collisions and damages rarely occur for most of the time. Therefore, in the first monitoring mode, the in-vehicle camera can reduce the frame rate and image resolution, and increase the frame rate and image resolution when further anomaly determination needs to be performed based on the second processing subunit, providing a basis for accurately judging whether an abnormal event occurred.

[0080] Exemplarily, in the first monitoring mode, the in-vehicle camera can be reduced from a high resolution (such as 4K) to a lower resolution (such as 1080P), and the image frame rate can be reduced from a high frame rate (such as 30fps) to a low frame rate (such as 5fps or 3fps).

[0081] Thus, after entering the parking monitoring mode, the in-vehicle camera runs at a low resolution and low frame rate for most of the time, further reducing power consumption.

[0082] After the controller instructs the in-vehicle camera to enter the second monitoring mode, the in-vehicle camera can increase the resolution and frame rate, acquire the monitoring video stream, and send the monitoring video stream to the controller. Exemplarily, the resolution is increased from 1080P to 4K, and the frame rate is increased from 5fps to 30fps.

[0083] After the controller receives the monitoring video stream, the second processing subunit of the in-vehicle SOC can be awakened, and the second processing subunit monitors anomalies based on the monitoring video stream.

[0084] Among them, the image processing ability of the second processing subunit is greater than that of the first processing subunit. When the first processing subunit cannot give a highly confident abnormal determination result, the second processing subunit deeply analyzes the monitored video stream based on its own image processing ability to determine whether an abnormal situation occurs.

[0085] Exemplarily, the first processing subunit can perform abnormal determination based on the image frames in a motion detection manner. For example, based on the frame difference method, calculate the absolute value of the pixel difference between consecutive frames for abnormal determination. Or, based on the optical flow method, calculate the motion vector of each pixel point in the image, and detect whether the area of the motion region and the motion trajectory meet the expectations, etc.

[0086] The second processing subunit can perform abnormal determination in a manner with higher accuracy. For example, adopt a pre-trained deep learning model to perform object detection and object tracking on the image frames to determine whether there are abnormal behaviors of the detected objects.

[0087] It should be noted that the above abnormal determination methods are only examples, and the embodiments of the present application do not limit this.

[0088] Adopting the solution of the embodiments of the present application, the controller first wakes up the first processing subunit of the in-vehicle SOC. When the determination result made by the first processing subunit is that an abnormality occurs, or when the first processing subunit cannot give a highly confident determination result, it instructs the in-vehicle camera to enter the second monitoring mode. The in-vehicle camera acquires the monitored video stream at a higher resolution and frame rate and sends it to the controller. The controller wakes up the second processing subunit, and the second processing subunit makes a more in-depth abnormal determination based on its stronger image processing ability compared to the first processing subunit to obtain a highly confident abnormal determination result.

[0089] It can be seen that it is possible to realize abnormal monitoring during the parking monitoring process, obtain a highly confident abnormal determination result, and reduce the probability of missing abnormal situations while saving power as much as possible.

[0090] In some embodiments of the present application, the in-vehicle camera performs image difference comparison on the reference image frame and the to-be-tested image frame, including: the in-vehicle camera performs pixel comparison on the reference image frame and the to-be-tested image frame to obtain difference pixels.

[0091] It can be understood that the image processing ability of the ISP chip of the in-vehicle camera is weak, and it is difficult to implement an image difference comparison method with high computational requirements. Therefore, the in-vehicle camera can adopt a pixel comparison method to directly compare the pixel values at the corresponding positions of the images to obtain the result of the difference comparison as difference pixels.

[0092] Correspondingly, the image difference satisfying the first condition indicates that the ratio of the difference pixels between the reference image frame and the image frame to be measured exceeds a preset threshold. Exemplarily, if the ratio of the difference pixels between the reference image frame and the image frame to be measured exceeds 30%, it is determined that the image difference satisfies the first condition.

[0093] By adopting the solution of the embodiment of the present application, since the image processing unit of the in-vehicle SOC can perform "fallback detection" on the image frame, the in-vehicle camera does not have to adopt an image difference comparison algorithm with high computational requirements and can be based only on pixel comparison. Thereby, the requirement for the image processing ability of the in-vehicle camera is reduced, and most in-vehicle cameras on the market can be adapted, improving the universality of the solution of the present application.

[0094] In some embodiments of the present application, the image frame set includes: the image frame of the first region of the reference image frame and the image frame of the first region of the image frame to be measured; wherein, the first region is the image region where the difference pixels between the reference image frame and the image frame to be measured are located.

[0095] Specifically, when the in-vehicle camera performs image difference comparison on the reference image frame and the image frame to be measured based on pixel comparison, when the ratio of the difference pixels between the reference image frame and the image frame to be measured exceeds the preset threshold, only the image frame of the first region of the reference image frame and the image frame of the first region of the image frame to be measured may be sent to the controller, and the first region is the image region where the difference pixels between the reference image frame and the image frame to be measured are located.

[0096] By adopting the solution of the embodiment of the present application, the in-vehicle camera does not have to adopt an image difference comparison algorithm with high computational requirements and can be based only on pixel comparison. And in necessary cases, the in-vehicle camera only needs to transmit the image frames of the image region where the difference pixels are located to the controller, further reducing the data transmission volume between the in-vehicle camera and the controller and reducing the energy consumption caused by data transmission.

[0097] For ease of understanding, the vehicle parking monitoring method provided by the embodiment of the present application will be further introduced below with reference to the accompanying drawings.

[0098] As Figure 4 shown, it includes the following steps: S401: Enter the parking monitoring mode.

[0099] S402: The camera module reduces the resolution to 1080P or lower, and the camera module reduces the frame rate to 5fps or lower.

[0100] Specifically, the controller instructs the camera module to enter the first monitoring mode, and the camera module reduces the resolution and the frame rate.

[0101] S403: The background of the low-frame-rate and low-resolution image is stored in the camera module, and the image difference comparison is processed in the SRAM of the camera module.

[0102] The camera module captures the background of the low-frame-rate and low-resolution image as a reference image frame. The comparison of image differences can be processed based on the SRAM.

[0103] S404: Transmit the low-resolution and low-frame-rate difference image or the high-resolution and high-frame-rate image through the virtual channel.

[0104] When the camera module detects a large difference between images, it transmits the low-resolution and low-frame-rate difference image to the controller through the virtual channel. In addition, when further needed, it sends the high-resolution and high-frame-rate image to the controller.

[0105] S405: After detecting the wake-up event, the SOC resumes to the normal mode.

[0106] After the controller wakes up the image processing unit of the vehicle-mounted SOC, the image processing unit performs further anomaly determination.

[0107] Adopting the solution of the embodiment of the present application, after the vehicle enters the parking monitoring mode, the image processing unit of the vehicle-mounted SOC is controlled to sleep, and the vehicle-mounted camera is instructed to enter the first monitoring mode. The vehicle-mounted camera performs difference comparison on the captured image frames and only sends the image frames to the controller when it is initially confirmed that the image difference is large. The controller then wakes up the image processing unit, and the image processing unit further analyzes the image frames.

[0108] Thus, the image frames are only sent to the controller when in-depth analysis of the image frames is required, greatly reducing the data transmission volume between the vehicle-mounted camera and the controller and reducing the energy consumption caused by data transmission.

[0109] In addition, in the first monitoring mode, the image processing unit of the vehicle-mounted SOC is in the sleep state most of the time, and is only woken up by the controller when the vehicle-mounted camera determines that the difference between the image frames is large. Compared with the image processing unit of the vehicle-mounted SOC always remaining awake, the energy consumption can be significantly reduced. The reduction of energy consumption can increase the cruising range of the vehicle.

[0110] The embodiment of the present application also provides a parking monitoring system for a vehicle, as Figure 5 shown, the parking monitoring system 500 of the vehicle includes: a controller 501, a vehicle-mounted SOC 502, and a vehicle-mounted camera 503.

[0111] Among them, the controller 501 is used to control the image processing unit of the vehicle-mounted SOC 502 to sleep and instruct the vehicle-mounted camera 503 to enter the first monitoring mode in response to the vehicle entering the parking monitoring mode; The in-vehicle camera 503 is used to obtain a reference image frame and a to-be-tested image frame in the first monitoring mode, compare the reference image frame and the to-be-tested image frame for image differences, and send an image frame set to the controller 501 when the image differences meet the first condition. The image frame set includes: the reference image frame and the to-be-tested image frame; The controller 501 is further configured to wake up the image processing unit in response to receiving the image frame set, so that the image processing unit performs anomaly monitoring based on the image frame set.

[0112] With the solution of the embodiment of the present application, the in-vehicle camera compares the collected image frames for differences and only sends the image frames to the controller when it is initially confirmed that the image differences are large. The controller then wakes up the image processing unit, and the image processing unit further analyzes the image frames. Thus, the image frames are only sent to the controller when in-depth analysis of the image frames is required, greatly reducing the data transmission volume between the in-vehicle camera and the controller and reducing the energy consumption caused by data transmission.

[0113] In the first monitoring mode, the image processing unit of the in-vehicle SOC is mostly in a sleep state, and is only woken up by the controller when the in-vehicle camera determines that the differences between the image frames are large. Compared with the image processing unit of the in-vehicle SOC always remaining in a wake-up state, the energy consumption can be significantly reduced. The reduction of energy consumption can increase the driving range of the vehicle.

[0114] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of methods. To implement the above functions, the parking monitoring device or electronic device of the vehicle includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraint conditions of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0115] Embodiments of the present application can divide functional modules according to the above method, for example, a parking monitoring device or an electronic device of a vehicle. For example, a parking monitoring device or an electronic device of a vehicle may include respective functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0116] Figure 6 is a block diagram of a parking monitoring device of a vehicle shown according to an exemplary embodiment. Referring to Figure 6 the parking monitoring device of the vehicle includes a control module 601 and a wake-up module 602.

[0117] The control module 601 is configured to, in response to the vehicle entering the parking monitoring mode, control the image processing unit of the in-vehicle SOC to enter the sleep state, and instruct the in-vehicle camera to enter the first monitoring mode; in the first monitoring mode, the in-vehicle camera acquires a reference image frame and a to-be-tested image frame, performs image difference comparison on the reference image frame and the to-be-tested image frame, and when the image difference meets the first condition, sends an image frame set to the in-vehicle SOC, where the image frame set includes: the reference image frame and the to-be-tested image frame; The wake-up module 602 is configured to, in response to receiving the image frame set, wake up the image processing unit so that the image processing unit performs abnormal monitoring based on the image frame set.

[0118] In a possible way, the image processing unit includes a first processing subunit and a second processing subunit; The wake-up module is specifically configured to: in response to receiving the image frame set, wake up the first processing subunit so that the first processing subunit performs abnormal determination based on the image frame set; The device further includes an indication module, configured to: in the case where the determination result of the abnormal determination shows an abnormality, or the confidence level of the determination result of the abnormal determination is less than a preset threshold, instruct the in-vehicle camera to enter the second monitoring mode; in the second monitoring mode, the in-vehicle camera acquires a monitoring video stream and sends the monitoring video stream to the in-vehicle SOC; The wake-up module is further configured to: in response to receiving the monitoring video stream, wake up the second processing subunit so that the second processing subunit performs abnormal monitoring based on the monitoring video stream.

[0119] In a possible way, the image processing capability and energy consumption of the first processing subunit are respectively less than those of the second processing subunit.

[0120] In one possible way, the frame rate and image resolution of the in-vehicle camera for collecting images in the first monitoring mode are respectively less than those of the in-vehicle camera for collecting images in the second monitoring mode.

[0121] In one possible way, the image difference satisfying the first condition indicates that the image difference evaluation value between the reference image frame and the image frame to be measured exceeds a preset evaluation threshold.

[0122] In one possible way, the image difference satisfying the first condition indicates that the proportion of the different pixels between the reference image frame and the image frame to be measured exceeds a preset threshold.

[0123] In one possible way, the set of image frames includes: the image frames of the first region of the reference image frame and the image frames of the first region of the image frame to be measured; wherein, the first region is the image region where the different pixels between the reference image frame and the image frame to be measured are located.

[0124] Figure 7 It is a block diagram of an electronic device shown according to an exemplary embodiment. As Figure 7 shown, the electronic device includes but is not limited to: a processor 701 and a memory 702.

[0125] Among them, the above-mentioned memory 702 is used to store the executable instructions of the above-mentioned processor 701. It can be understood that the above-mentioned processor 701 is configured to execute instructions to implement the vehicle energy consumption calculation method in the above-mentioned embodiment.

[0126] It should be noted that those skilled in the art can understand that Figure 7 the structure of the electronic device shown in Figure 7 does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than

[0127] shown, or combine some components, or have different component arrangements.

[0128] The memory 702 can be used to store software programs and various data. The memory 702 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required by at least one functional module (such as a determination unit, a processing unit, etc.). In addition, the memory 702 can include high-speed random access memory and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0129] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as the memory 702 including instructions. The above instructions can be executed by the processor 701 of the electronic device to implement the method in the above embodiment.

[0130] Optionally, the computer-readable storage medium can be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0131] In an exemplary embodiment, an embodiment of the present application also provides a computer program product including one or more instructions. The one or more instructions can be executed by the processor of the electronic device to complete the method in the above embodiment.

[0132] It should be noted that when the instructions in the above computer-readable storage medium or the one or more instructions in the computer program product are executed by the processor of the electronic device, each process of the above method embodiment is implemented, and the same technical effects as the above method can be achieved. To avoid repetition, it will not be elaborated here.

[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0134] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units, that is, they can be located in one place, or they can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks or optical discs that can store program codes.

[0138] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A parking monitoring method for a vehicle, characterized in that, A controller applied to the vehicle, the method comprising: In response to the vehicle entering the parking monitoring mode, controlling the image processing unit of the on-vehicle system-on-chip (SOC) to sleep, and instructing the on-vehicle camera to enter the first monitoring mode; in the first monitoring mode, the on-vehicle camera acquires a reference image frame and a to-be-tested image frame, performs image difference comparison on the reference image frame and the to-be-tested image frame, and when the image difference meets the first condition, sends an image frame set to the controller, the image frame set including: the reference image frame and the to-be-tested image frame; In response to receiving the image frame set, waking up the image processing unit so that the image processing unit performs anomaly monitoring based on the image frame set.

2. The parking monitoring method of a vehicle according to claim 1, characterized in that, The image processing unit includes a first processing subunit and a second processing subunit; The waking up the image processing unit in response to receiving the image frame set includes: In response to receiving the image frame set, waking up the first processing subunit so that the first processing subunit performs anomaly determination based on the image frame set; The method further includes: In the case where the determination result of the anomaly determination shows an anomaly, or the confidence level of the determination result of the anomaly determination is less than a preset threshold, instructing the on-vehicle camera to enter the second monitoring mode; in the second monitoring mode, the on-vehicle camera acquires a monitoring video stream and sends the monitoring video stream to the controller; In response to receiving the monitoring video stream, waking up the second processing subunit so that the second processing subunit performs anomaly monitoring based on the monitoring video stream.

3. The parking monitoring method for a vehicle according to claim 2, wherein: The image processing capability and energy consumption of the first processing subunit are respectively less than those of the second processing subunit.

4. The parking monitoring method for a vehicle according to claim 2, wherein: The frame rate and image resolution of the on-vehicle camera for acquiring images in the first monitoring mode are respectively less than those of the on-vehicle camera for acquiring images in the second monitoring mode.

5. The parking monitoring method of a vehicle according to claim 1, wherein The image difference meeting the first condition indicates that the image difference evaluation value between the reference image frame and the to-be-tested image frame exceeds a preset evaluation threshold.

6. The parking monitoring method of a vehicle according to claim 1, wherein, The on-vehicle camera performing image difference comparison on the reference image frame and the to-be-tested image frame includes: The on-vehicle camera performs pixel comparison on the reference image frame and the to-be-tested image frame to obtain difference pixels.

7. The parking monitoring method for a vehicle according to claim 6, wherein: The image difference meeting the first condition indicates that the proportion of the difference pixels between the reference image frame and the to-be-tested image frame exceeds a preset threshold.

8. The parking monitoring method of a vehicle according to claim 7, characterized in that, The image frame set includes: the image frame of the first region of the reference image frame and the image frame of the first region of the to-be-tested image frame; wherein, the first region is the image region where the difference pixels between the reference image frame and the to-be-tested image frame are located.

9. A parking monitoring device for a vehicle, characterized in that, A controller applied to the vehicle, the device comprising: A control module, configured to control the image processing unit of the in-vehicle SOC to enter a sleep state and instruct the in-vehicle camera to enter a first monitoring mode in response to the vehicle entering the parking monitoring mode; in the first monitoring mode, the in-vehicle camera acquires a reference image frame and a to-be-tested image frame, performs image difference comparison on the reference image frame and the to-be-tested image frame, and sends an image frame set including the reference image frame and the to-be-tested image frame to the in-vehicle SOC when the image difference meets a first condition. A wake-up module, configured to wake up the image processing unit in response to receiving the image frame set, so that the image processing unit performs abnormal monitoring based on the image frame set.

10. The parking monitoring device for a vehicle according to claim 9, characterized in that, The image processing unit includes a first processing subunit and a second processing subunit. Specifically, the wake-up module is configured to: Wake up the first processing subunit in response to receiving the image frame set, so that the first processing subunit performs abnormal determination based on the image frame set. The device further includes an indication module, configured to: In the case where the determination result of the abnormal determination shows an abnormality, or the confidence level of the determination result of the abnormal determination is less than a preset threshold, instruct the in-vehicle camera to enter a second monitoring mode; in the second monitoring mode, the in-vehicle camera acquires a monitoring video stream and sends the monitoring video stream to the in-vehicle SOC. The wake-up module is further configured to: Wake up the second processing subunit in response to receiving the monitoring video stream, so that the second processing subunit performs abnormal monitoring based on the monitoring video stream.

11. A parking monitoring system for a vehicle, characterized in that, The parking monitoring system includes: a controller, an in-vehicle SOC, and an in-vehicle camera. The controller is configured to control the image processing unit of the in-vehicle SOC to enter a sleep state and instruct the in-vehicle camera to enter a first monitoring mode in response to the vehicle entering the parking monitoring mode. The in-vehicle camera is configured to acquire a reference image frame and a to-be-tested image frame in the first monitoring mode, perform image difference comparison on the reference image frame and the to-be-tested image frame, and send an image frame set including the reference image frame and the to-be-tested image frame to the controller when the image difference meets a first condition. The controller is further configured to wake up the image processing unit in response to receiving the image frame set, so that the image processing unit performs abnormal monitoring based on the image frame set.

12. A vehicle, characterized in that, The vehicle includes the parking monitoring device of the vehicle according to claim 9 or 10.

Images