Vehicle monitoring awakening method and device

Abstract

The invention provides a vehicle monitoring awakening method and device.The method comprises the steps that an MCU receives infrared collection information sent by an infrared collection module in real time, whether an AI module is awakened or not is determined according to the infrared collection information, if yes, the AI module is awakened, and if not, the AI module is awakened; the MCU determines whether to wake up the SoC according to a heat source identification result obtained by the AI module according to the infrared acquisition information and the infrared acquisition information, if yes, the SoC is awakened, the SoC initializes the image acquisition module and sets the frame rate of the image acquisition module as a first frame rate, the SoC determines whether to execute monitoring processing by the server according to the image, and if yes, the SoC is awakened. And if not, setting the frame rate of the image acquisition module as a second frame rate, activating the communication module, and sending the image sequence to the server through the communication module to execute monitoring processing. According to the parking monitoring method and device, power consumption is distributed according to needs in a step-by-step awakening mode, the standby time of the battery is prolonged, and the sustainability of parking monitoring is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle monitoring technology, and in particular to a vehicle monitoring wake-up method and device. Background Art

[0002] With the development of society, the number of vehicles continues to rise, the use scenarios of vehicles continue to expand, and the overlap between parking areas and areas where people and animals move is becoming increasingly frequent. When a vehicle is stationary or turned off, the driver often leaves the vehicle, and the surrounding environment of the vehicle is unmonitored. If pedestrians, children or animals are active around the vehicle, it is very likely that safety accidents such as collisions and squeezing will occur due to their proximity to the vehicle, posing a serious threat to the life safety of people and animals, and also causing property losses to the owner of the vehicle. Therefore, a technology that can effectively detect moving people or animals when the vehicle is stationary or turned off has become an urgent need to improve the level of vehicle safety protection and reduce the occurrence of accidents.

[0003] In the prior art, vehicle environment information is usually collected through traditional sensors or cameras operating in a full-power state, and a microcontroller unit (MCU) determines whether there is a mobile heat source around the vehicle based on the collected information.

[0004] However, traditional sensors in the existing technology cannot detect changes in heat sources, resulting in the need to frequently wake up the camera once a heat source appears. The camera is in full power consumption mode, so the power consumption of the entire device is high, which affects battery life. Summary of the Invention

[0005] The purpose of this application is to provide a vehicle monitoring wake-up method and device to address the deficiencies in the above-mentioned prior art, so as to solve the problem of high power consumption of the entire device during the monitoring process in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows: In a first aspect, the present application provides a vehicle monitoring wake-up method, the method comprising: The microcontroller unit MCU receives the infrared acquisition information sent by the infrared acquisition module in real time, and determines whether to wake up the artificial intelligence AI module in the MCU according to the infrared acquisition information; If so, the AI ​​module is awakened, and the AI ​​module obtains a heat source identification result according to the infrared collection information, and the MCU determines whether to wake up the system-on-chip SoC according to the heat source identification result of the AI ​​module and the infrared collection information; If so, waking up the SoC, initializing the image acquisition module by the SoC and setting the frame rate of the image acquisition module to the first frame rate, and determining by the SoC whether to execute monitoring processing by the server based on the image sequence acquired by the image acquisition module; If so, the frame rate of the image acquisition module is adjusted to a second frame rate, and the communication module is activated to send the image sequence acquired by the image acquisition module to the server through the communication module to perform monitoring processing, wherein the second frame rate is greater than the first frame rate.

[0007] Optionally, determining whether to wake up the AI ​​module in the MCU according to the infrared collection information includes: determining, based on the infrared collected information, whether a change amplitude of the heat source is greater than an amplitude threshold, and whether a first residence time of the heat source in the detection area is greater than a first residence time threshold; If the change amplitude of the heat source is greater than the amplitude threshold, and the first residence time of the heat source in the detection area is greater than the first residence time threshold, it is determined to wake up the AI ​​module in the MCU.

[0008] Optionally, determining whether to wake up the SoC according to the heat source identification result of the AI ​​module and the infrared collection information includes: determining a second residence time of the heat source in the detection area based on the infrared collected information; Determine whether to wake up the SoC according to the heat source identification result and the second stay time.

[0009] Optionally, determining whether to wake up the SoC according to the heat source identification result and the second dwell time includes: If the heat source identification result of the AI ​​module is that there is a target in the detection area and the confidence of the heat source identification result is greater than the confidence threshold, or if the second stay time is greater than the second stay time threshold, or the area of ​​the target identified by the AI ​​module is greater than the preset area threshold, it is determined to wake up the SoC.

[0010] Optionally, determining whether the server performs monitoring processing according to the image sequence captured by the image capture module includes: determining a third residence time of the target in the detection area according to the image sequence; determining a distance between a target and a vehicle based on the image sequence; determining a moving trajectory of a target according to the image sequence; predicting a target behavior label based on the image sequence, wherein the behavior label is used to indicate whether the target is suspicious; Determine whether to perform monitoring processing by the server according to the third stay duration, the distance, the movement trajectory, and the behavior tag.

[0011] Optionally, determining whether to perform monitoring processing by the server based on the third stay duration, the distance, the movement trajectory, and the behavior tag includes: Determining whether the third stay duration is greater than a third stay duration threshold; determining whether the distance is less than a distance threshold; Determining, based on the movement trajectory, whether the target is performing a target action or wandering within the key area; Predicting whether the target may exhibit suspicious behavior based on the behavior label; If the third stay duration is greater than the third stay duration threshold, or the distance is less than the distance threshold, or the target performs a target action or wanders around in the key area, or the target may exhibit suspicious behavior, it is determined that the server performs monitoring processing.

[0012] Optionally, determining the movement trajectory of the target according to the image sequence includes: The spatial movement path of the target in the image sequence is reconstructed based on a tracking algorithm to obtain the movement trajectory of the target.

[0013] Optionally, predicting a behavior label of a target based on the image sequence includes: Based on convolutional neural networks and recurrent neural networks, behavior classification is performed on the image sequence to obtain behavior labels of the targets.

[0014] Optionally, the method further includes: If the third stay duration is not greater than the third stay duration threshold, and the distance is not less than the distance threshold, and the target does not perform target actions or wander around in the key area, and the target does not exhibit suspicious behavior, the SoC is put to sleep.

[0015] In a second aspect, the present application provides a vehicle monitoring wake-up device, which is used to execute the vehicle monitoring wake-up method as described in the first aspect.

[0016] The beneficial effects of the present application are: the MCU determines whether to wake up the AI ​​module in the MCU based on the infrared acquisition information sent by the infrared acquisition module. If so, the AI ​​module is awakened, and the SoC is determined based on the heat source recognition result of the AI ​​module and the infrared acquisition information. If so, the SoC is awakened, the SoC initializes the image acquisition module and sets the frame rate of the image acquisition module to the second frame rate, and determines whether the monitoring processing is performed by the server based on the image sequence collected by the image acquisition module. If so, the frame rate of the image acquisition module is adjusted to the second frame rate, and the communication module is activated, and the image sequence collected by the image acquisition module is sent to the server through the communication module to perform monitoring processing. In the above method, the low-power MCU first determines whether to wake up the higher-power AI module in the MCU, and then determines whether to wake up the high-power SoC. After the SoC is awakened, the image acquisition module is first enabled to acquire images at a low frame rate. If the server needs to perform monitoring processing, the image acquisition module is enabled to acquire images at a high frame rate. It can be seen that in the monitoring and wake-up process of this embodiment, the low-power module is first monitored, and when the conditions are met, the high-power module is gradually awakened for further monitoring, thereby realizing the energy consumption optimization configuration of the monitoring and wake-up system at different risk levels, and forming a link of multimodal perception and progressive judgment. This power consumption allocation method on demand greatly extends the standby time of the battery and improves the sustainability of parking monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a schematic diagram of an application scenario of a vehicle monitoring wake-up method provided in an embodiment of the present application; Figure 2 This is a flow chart of a vehicle monitoring wake-up method provided in an embodiment of the present application; Figure 3 This is a flowchart of determining whether a monitoring process is performed by a server, provided by an embodiment of the present application; Figure 4 This is another flowchart of determining whether to perform monitoring processing by the server provided in an embodiment of the present application; Figure 5 This is a flow chart of another vehicle monitoring wake-up method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0020] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0022] In existing technologies, vehicle environmental information is collected through traditional sensors or cameras operating at full power consumption. The MCU then uses this information to determine whether there are moving heat sources around the vehicle. However, these sensors are unable to identify the type of heat source or its changes. Once a heat source appears, the camera must be frequently awakened, and the camera, operating at full power consumption, results in poor detection and high overall power consumption, impacting battery life.

[0023] Based on this, this application proposes a vehicle monitoring wake-up method. The MCU wakes up the artificial intelligence (AI) module in the MCU based on infrared acquisition information, and wakes up the system-on-a-chip (SoC) based on the heat source identification results and infrared acquisition information obtained by the AI ​​module. The SoC initializes the image acquisition module and determines whether the server performs monitoring processing based on the acquired image. If so, the image acquisition module's acquisition frame rate is increased and the image sequence is sent to the server. In the above steps, the vehicle's surrounding environment is detected step by step through the MCU, the AI ​​module in the MCU, and the SoC. The operating power of the AI ​​module in the MCU is higher than that of the MCU, and the operating power of the SoC is higher than that of the AI ​​module in the MCU. Therefore, by waking up and detecting the surrounding environment step by step, the system power consumption can be reduced while ensuring detection accuracy.

[0024] Next, refer to Figure 1 The vehicle and its application scenarios using the vehicle monitoring wake-up method are introduced. Figure 1 This is a schematic diagram of an application scenario of a vehicle monitoring wake-up method provided in an embodiment of the present application.

[0025] like Figure 1 As shown, a monitoring and wake-up system is deployed in the vehicle. The system includes an infrared acquisition module, an MCU, a SoC, and an image acquisition module. The MCU is connected to the infrared acquisition module and the SoC, which is also connected to the image acquisition module. The MCU and SoC can be connected via a vehicle-standard communication bus. For example, the vehicle-standard communication bus can be a Controller Area Network with Flexible Data Rate (CAN FD) bus, Automotive Ethernet (Automotive Ethernet), or a Local Interconnect Network (LIN) bus. The SoC can connect to a server to send information such as images.

[0026] A detection area can be pre-set around the vehicle. The infrared acquisition module can collect infrared information in the detection area and send it to the MCU. The image acquisition module can collect images in the detection area and send them to the SoC.

[0027] It is worth noting that the vehicle monitoring wake-up method can be applied to monitor heat sources in the vehicle's detection area when the vehicle is stationary or turned off. The detection area can be a pre-set area around the vehicle that needs to be detected.

[0028] Next, refer to Figure 2The specific implementation of the vehicle monitoring wake-up method is introduced. Figure 2 This is a flow chart of a vehicle monitoring wake-up method provided in an embodiment of the present application.

[0029] S201. The MCU receives infrared acquisition information sent by the infrared acquisition module in real time, and determines whether to wake up the AI ​​module in the MCU based on the infrared acquisition information.

[0030] Alternatively, the infrared acquisition module may be a passive infrared sensor that detects heat sources around the vehicle by detecting background temperature differences, and generates infrared acquisition information that is sent to the MCU. After acquiring the initial infrared data, the infrared acquisition module may perform pre-processing such as filtering and amplification to obtain infrared acquisition information, which is then sent to the MCU.

[0031] As an optional implementation, the infrared acquisition information may be a thermal image sequence or a grayscale image sequence.

[0032] Optionally, the MCU can determine the change amplitude of the heat source and the residence time of the heat source based on the infrared collection information, and thus use the change amplitude of the heat source and the residence time of the heat source as preconditions to determine whether to wake up the AI ​​module.

[0033] Optionally, the power consumption of the MCU after waking up the AI ​​module is higher than the power consumption of the MCU when it is working. Before waking up the AI ​​module, the AI ​​module is in a dormant state.

[0034] S202: If yes, wake up the AI ​​module, and the AI ​​module obtains the heat source identification result according to the infrared collection information. The MCU determines whether to wake up the SoC according to the heat source identification result of the AI ​​module and the infrared collection information.

[0035] Optionally, the power consumption of the SoC is higher than that of the MCU and the power consumption when the MCU wakes up the AI ​​module. The SoC may be an Android SoC. Exemplarily, when the AI ​​module is woken up, the total power consumption is less than 5 milliwatts.

[0036] Optionally, the AI ​​module has self-learning capabilities and can identify complex human outlines and behavioral characteristics based on massive training data, which is far superior to a single trigger mechanism based on heat sources or heat source movement. Heat source identification can be performed on the infrared collected information to obtain a heat source identification result. The heat source identification result may include heat source type information, heat source residence time, and heat source area. The heat source type information may include, for example, the probability that the heat source is a person, dog, or flying insect. The heat source residence time may be the length of time the heat source remains in the detection area. The heat source area may be the area occupied by the heat source within the detection area.

[0037] Optionally, the MCU can determine whether it is necessary to wake up the SoC based on the heat source identification results, infrared collection information, and preset judgment conditions. As an optional implementation, the MCU can make judgments from multiple angles based on the heat source identification results, infrared collection information, and preset judgment conditions. If one of the judgments meets the conditions, the SoC is woken up. The judgment can be, for example: judging whether the heat source type meets the conditions based on the heat source type information in the heat source identification results and the preset type conditions, and judging whether the heat source stays for a long time based on the heat source stay time and the preset time condition, and judging whether the heat source is a large-area heat source based on the heat source area and the preset area condition.

[0038] S203: If yes, wake up the SoC, initialize the image acquisition module by the SoC and set the frame rate of the image acquisition module to the first frame rate, and determine whether the server performs monitoring processing according to the image sequence acquired by the image acquisition module.

[0039] Optionally, the image acquisition module may be an on-board camera. The image acquisition module may acquire images according to different frame rate requirements. The higher the frame rate requirement, the higher the power consumption. As an optional embodiment, the image acquisition module may further set the resolution of the image acquisition module to a first resolution, so that the image acquisition module acquires images according to the first frame rate and the first resolution. For example, the first frame rate may be 10 frames per second, and the first resolution may be a resolution with a vertical resolution of 720 pixels.

[0040] Optionally, after waking up the SoC, the SoC wakes up the image acquisition module to start working. Before waking up the SoC, the SoC is in a dormant state, and before the SoC wakes up the image acquisition module, the image acquisition module is in a power-off or dormant state.

[0041] Optionally, the SoC may wake up and configure the image acquisition module via an Inter-Integrated Circuit (IIC) bus. The configuration content may include resolution, frame rate, etc.

[0042] Optionally, the SoC can perform image recognition based on the image sequence captured by the image acquisition module and determine whether the server should perform monitoring processing based on the image recognition results. The image recognition results can include: whether a target exists, the target's residence time, the target's movement trajectory, and the target's distance. The target is the entity that the recognition algorithm needs to detect, classify, or track.

[0043] Alternatively, if the server needs to perform monitoring processing, it means that a heat source that needs attention appears in the detection area of ​​the current vehicle. For example, a person who stays in the detection area of ​​the vehicle for more than 1 minute appears.

[0044] S204. If yes, adjust the frame rate of the image acquisition block to a second frame rate, activate the communication module, and send the image sequence acquired by the image acquisition module to the server through the communication module to perform monitoring processing, wherein the second frame rate is greater than the first frame rate.

[0045] Optionally, if the server needs to perform monitoring processing, the SoC reconfigures the image acquisition module via the IIC. The configuration includes: a second frame rate. Optionally, the configuration may also include: a second resolution. The second resolution is greater than the first resolution. For example, the second frame rate may be 30 frames per second, and the second resolution may be a vertical resolution of 1080 pixels.

[0046] Optionally, the image sequence is composed of a plurality of images in a time sequence. The image sequence may be a video, for example.

[0047] Optionally, if monitoring processing is required by the server, the SoC activates the communication module. The communication module can, for example, be a 4th Generation Mobile Communication Technology (4G) or 5th Generation Mobile Communication Technology (5G) cellular network. The communication module can encrypt the images or image sequences captured at the second frame rate using Transport Layer Security (TLS) and transmit them to the server via Over-the-Air (OTA) technology for monitoring processing. This processing may include analyzing the current vehicle environment to obtain a suspicious judgment result. The server can store the image sequence and the corresponding suspicious judgment result, and transmit the suspicious judgment result to the user's mobile device. The suspicious judgment result can indicate whether a suspicious situation has occurred, including the degree of risk.

[0048] As an optional implementation, the communication module may transmit not only the image or image sequence but also the timestamp and geographic location information.

[0049] In addition, the communication module can also use OTA to send the image or image sequence collected based on the second frame rate to the user's mobile device, and the server can send the suspicious judgment result obtained based on the obtained image sequence to the mobile device, and the mobile device, such as a smart phone with a specific application installed, can realize real-time monitoring notification, remote viewing, voice alarm and manual setting and other functions. Specifically, the user can use the mobile device to monitor the heat source and monitoring conditions in the detection area of ​​the vehicle in real time. When the suspicious judgment result sent by the server indicates a suspicious situation, the mobile device can generate a real-time monitoring notification and a voice alarm based on the suspicious judgment result to remind the user. In addition, the user can use the mobile device to set the parameters used in the execution of the vehicle monitoring wake-up method, including the first frame rate, the second frame rate, etc. Based on this, the user experience is improved, and a full-link closed loop from vehicle event detection to user remote perception processing is realized.

[0050] Optionally, if monitoring processing needs to be performed by the server, the system enters a fully operational state and power consumption is increased to a normal operating level, ensuring timely response capabilities and high-definition image evidence collection capabilities when the vehicle may be threatened.

[0051] In this embodiment, the MCU determines whether to wake up the AI ​​module in the MCU based on the infrared acquisition information sent by the infrared acquisition module. If so, the AI ​​module is awakened, and the SoC is determined based on the heat source recognition result of the AI ​​module and the infrared acquisition information. If so, the SoC is awakened, the SoC initializes the image acquisition module and sets the frame rate of the image acquisition module to the second frame rate, and determines whether the monitoring processing is performed by the server based on the image sequence collected by the image acquisition module. If so, the frame rate of the image acquisition module is adjusted to the second frame rate, and the communication module is activated. The image sequence collected by the image acquisition module is sent to the server through the communication module to perform monitoring processing. In the above method, the low-power MCU first determines whether to wake up the higher-power AI module in the MCU, and then determines whether to wake up the high-power SoC. After the SoC is awakened, the image acquisition module is first enabled to acquire images at a low frame rate. If the server needs to perform monitoring processing, the image acquisition module is enabled to acquire images at a high frame rate. It can be seen that in the monitoring and wake-up process of this embodiment, the low-power module is first monitored, and when the conditions are met, the high-power module is gradually awakened for further monitoring, thereby realizing the energy consumption optimization configuration of the monitoring and wake-up system at different risk levels, and forming a link of multimodal perception and progressive judgment. This power consumption allocation method on demand greatly extends the standby time of the battery and improves the sustainability of parking monitoring.

[0052] Next, a feasible method for determining whether to wake up the AI ​​module in the MCU according to the infrared collected information in the above step S201 is introduced.

[0053] Optionally, based on the infrared collection information, it is determined whether the change amplitude of the heat source is greater than an amplitude threshold, and whether the first residence time of the heat source in the detection area is greater than a first residence time threshold.

[0054] Optionally, the MCU can perform temperature difference analysis and motion trajectory analysis on the infrared collected information to determine the heat source, thereby determining the change amplitude of the heat source and the first residence time of the heat source in the detection area. The change amplitude can be the movement distance of the heat source.

[0055] Optionally, the amplitude threshold and the first stay duration threshold are preset values, and the user can set the amplitude threshold and the first stay duration threshold through a mobile device.

[0056] Optionally, if the change amplitude of the heat source is greater than the amplitude threshold, and the first residence time of the heat source in the detection area is greater than the first residence time threshold, it is determined to wake up the AI ​​module in the MCU.

[0057] As an optional implementation, to prevent high-power modules from being frequently woken up, additional conditions can be added to determine whether to wake up the AI ​​module in the MCU. For example, if the amplitude of the heat source change is greater than the amplitude threshold, the first residence time of the heat source in the detection area is greater than the first residence time threshold, and the residence area of ​​the heat source is greater than the residence area threshold, then the AI ​​module in the MCU is determined to be woken up.

[0058] In this embodiment, it is determined based on the infrared acquisition information whether the change amplitude of the heat source is greater than the amplitude threshold, and whether the first dwell time is greater than the first dwell time threshold, to determine whether to wake up the AI ​​module in the MCU. In this stage of the embodiment, only the infrared acquisition module and the MCU are working, and there is no need for image acquisition and complex image recognition operations, which has the advantages of extremely low power consumption and fast response.

[0059] After determining to wake up the AI ​​module in the MCU, the following describes the specific steps of determining whether to wake up the SoC based on the heat source identification result and infrared collection information of the AI ​​module in step S202.

[0060] Optionally, a second residence time of the heat source in the detection area is determined based on the infrared collection information.

[0061] As an optional embodiment, the second dwell time may be the sum of the first dwell time of the heat source in the detection area, as counted when the MCU determines to wake up the AI ​​module, and the dwell time of the heat source counted after the AI ​​module is woken up. For example, if, when the MCU determines whether to wake up the AI ​​module, the heat source's first dwell time in the detection area is detected to be 3 seconds, and the heat source has remained in the detection area for 1 second since the AI ​​module was woken up until the current moment, the second dwell time is 4 seconds.

[0062] Optionally, whether to wake up the SoC is determined according to the heat source identification result and the second stay time.

[0063] As an optional implementation, the heat source identification result may be a heat source type. Based on this, when the heat source type is a preset type, or the second dwell time is greater than a second dwell time threshold, it is determined to wake up the SoC.

[0064] As another optional implementation, the heat source identification result may also be the heat source area. Based on this, when the heat source area is greater than a preset area threshold, or the second dwell time is greater than the second dwell time threshold, it is determined to wake up the SoC.

[0065] In this embodiment, the second dwell time of the heat source is determined based on the infrared information collected. The SoC is then determined based on the heat source identification result and the second dwell time. Because the heat source identification result, determined by the AI ​​module, is used to determine whether to wake the SoC, the environmental risk level is accurately determined, eliminating the need for image acquisition and complex image recognition operations, resulting in extremely low power consumption and fast response times.

[0066] Furthermore, an implementation method of determining whether to wake up the SoC according to the heat source identification result and the second dwell time in the above steps is introduced.

[0067] Optionally, if the heat source identification result of the AI ​​module is that there is a target in the detection area and the confidence of the heat source identification result is greater than the confidence threshold, or if the second stay time is greater than the second stay time threshold, or the area of ​​the target identified by the AI ​​module is greater than the preset area threshold, it is determined to wake up the SoC.

[0068] Optionally, the AI ​​module can be equipped with a lightweight AI inference engine, such as a neural network model with the third generation lightweight mobile neural network (MobileNetV3-Tiny) or efficient neural network (EfficientNet-lite) as the core.

[0069] Optionally, the infrared information is fed into the AI ​​module, which first extracts features from it through multiple convolutional layers. As a core component of the deep learning model, the convolutional layer contains learnable convolution kernels, with sizes such as 3×3 or 5×5. These kernels move pixel by pixel across the image in a sliding window fashion, extracting local features from the infrared information through convolution operations.

[0070] The size and resolution of the feature map can be changed by adjusting the step size and padding method of the convolution layer and introducing the pooling layer. Small-scale feature maps retain the global semantic information of the image, while large-scale feature maps focus on local details. After completing feature extraction, the AI ​​module inputs the extracted features into the classification subnetwork, namely the classification head. The classification head usually consists of a fully connected layer and a Softmax activation function. The function of the fully connected layer is to convert the feature map output by the convolution layer into multiple sets of vectors, and through a large number of neurons and weight connections, further abstract and integrate the features and map them to the dimensional space of the target category. For example, if the AI ​​module needs to recognize 10 predefined categories, such as "human," "vehicle," and "animal," the number of output neurons in the fully connected layer is 10. The Softmax activation function then processes the output of the fully connected layer, converting it into a probability distribution vector. The Softmax function compresses each output value to a range between 0 and 1, with the sum of all values ​​being 1. Each value represents the predicted probability of the corresponding category. For example, the output vector [0.1, 0.8, 0.05, 0.05] indicates that the model believes the image has an 80% probability of belonging to the second category and an 80% confidence level, while the probability of belonging to other categories is relatively low. If the predicted probability of a category is greater than the confidence threshold, it is determined that an object is present in the image and belongs to that target category.

[0071] As an optional implementation, each classification category can correspond to a confidence threshold. When the confidence level of the heat source identification result is greater than the confidence threshold of any category, it can be determined that the confidence level of the heat source identification result is greater than the confidence threshold. As another optional implementation, a confidence threshold can be set. If the confidence level of the heat source identification result exceeds the confidence threshold, it indicates that a target exists within the detection area.

[0072] Optionally, if the target moves within the detection area, the area of ​​the target identified by the AI ​​module is the maximum area of ​​the target within the detection area during the movement. The area of ​​the identified target is used to exclude non-objects of interest such as small animals.

[0073] In this embodiment, the confidence of the target and heat source identification results in the heat source identification results of the AI ​​module, or the second stay time, or the area of ​​the target is judged to accurately determine the degree of environmental risk. There is no need for image acquisition and complex image recognition operations, and it has the advantages of extremely low power consumption and fast response.

[0074] After waking up the SoC, refer to Figure 3 The process of determining whether the server should perform monitoring processing based on the image sequence collected by the image acquisition module is introduced. Figure 3This is a flowchart of determining whether monitoring processing is performed by a server, provided in an embodiment of the present application.

[0075] S301: Determine a third stay time of the target in the detection area according to the image sequence.

[0076] As an optional embodiment, the third dwell time may be the sum of the second dwell time and the duration of the target's stay in the detection area, as determined based on the image sequence after the SoC is awakened. For example, if the second dwell time is 10 seconds and, after the SoC is awakened, the target's stay in the detection area is determined to be 2 seconds, the third dwell time is 12 seconds.

[0077] As another optional implementation, the third dwell time is the dwell time of directly identifying and determining the target according to the image sequence after waking up the SoC.

[0078] S302: Determine the distance between the target and the vehicle based on the image sequence.

[0079] Optionally, the SoC can extract and match image features of the image sequence in sequence based on a deep learning model to generate a depth map. Finally, combined with the parameters of the image acquisition module and the scene scale, the pixel values ​​in the depth map are converted into actual physical distances to obtain the distance between the target and the vehicle.

[0080] S303: Determine the target's moving trajectory based on the image sequence.

[0081] Optionally, the SoC can use target detection models such as YOLO and Faster Region-based Convolutional Neural Network (Faster R-CNN) to perform target detection, inter-frame matching, and trajectory association and optimization on image sequences, and output the target's movement trajectory.

[0082] Specifically, the target detection model is first used to scan each frame of the image sequence, identify the target, and output its location information. Next, a data association algorithm is used to establish target correspondences between frames. Specifically, similarity is calculated using the target's position, velocity, and appearance features, matching the same target in adjacent frames. Deep learning then uses architectures such as twin networks and attention-based neural network architectures (Transformers) to predict the probability of cross-frame target associations. Finally, a trajectory is constructed based on the matching results, and optimization methods such as Kalman filtering and the Hungarian algorithm are used to smooth the trajectory and eliminate deviations caused by jitter or mismatching. The final output is the target's movement trajectory within the image sequence.

[0083] S304: predicting a behavior label of the target based on the image sequence, where the behavior label is used to indicate whether the target is suspicious.

[0084] Alternatively, SoC can use models such as YOLO to sequentially complete target detection and trajectory extraction, spatiotemporal feature encoding, and behavior modeling and classification based on image sequences to predict the target's behavior label. A behavior label is a symbolic or numerical identifier that quantifies, describes, and categorizes the target's behavioral attributes.

[0085] Specifically, the image sequence is input into models such as YOLO, which are used to detect the target and track the trajectory to obtain position and motion information. Then, dual-stream CNN, 3D CNN, etc. are used to extract the target's spatial appearance, temporal motion and other features. Finally, a classifier is built based on the deep learning model, combining multi-dimensional scores such as interactive behavior and appearance features to output the behavior confidence, which is then compared with the threshold to determine the target's behavior label.

[0086] It is worth noting that a large number of image sequences of suspicious objects can be used in advance to train the model. The suspicious object may be, for example, "two people fighting."

[0087] S305: Determine whether the server performs monitoring processing based on the third stay duration, distance, movement trajectory, and behavior tag.

[0088] Optionally, the third stay duration, distance, movement trajectory and behavior tag may be used as four judgment bases and judged separately. If there is a judgment that meets the conditions, it is determined that the server performs monitoring processing.

[0089] As an optional implementation, the area of ​​the target can be determined based on the image sequence, and whether the server should perform monitoring processing can be determined based on the third dwell time, distance, movement trajectory, behavior tag, and area. If the area exceeds the area threshold, it indicates that there may be an obstruction in the image acquisition module's acquisition area, and the image sequence needs to be uploaded to the server and the user's mobile device for processing.

[0090] In this embodiment, the third dwell time, distance, and movement trajectory are determined based on the image sequence, and the target's behavior label is predicted to determine whether the server performs monitoring processing. This multi-condition judgment method can reduce the false alarm rate.

[0091] Next, refer to Figure 4 The specific steps of determining whether the server performs monitoring processing according to the third stay time, distance, movement trajectory and behavior tag in step S305 are introduced. Figure 4 This is another flowchart provided by an embodiment of the present application for determining whether monitoring processing is performed by the server.

[0092] S401: Determine whether the third stay duration is greater than a third stay duration threshold.

[0093] Optionally, the user may pre-set a third stay duration threshold via the mobile terminal, wherein the third stay duration threshold is greater than the second stay duration threshold.

[0094] S402: Determine whether the distance is less than a distance threshold.

[0095] Optionally, the user may pre-set the distance threshold via the mobile terminal.

[0096] S403: Determine whether the target is performing a target action or wandering in the key area based on the movement trajectory.

[0097] Optionally, the key area may be a pre-set area that may threaten vehicle safety, such as a door handle area, etc. The target action performed in the key area may be an action that threatens vehicle safety, such as a door opening action, a throwing or placing action.

[0098] Optionally, if the movement trajectory indicates that the user repeatedly moves around in a certain area within a certain period of time, the target is determined to be wandering.

[0099] S404: Predict whether the target may exhibit suspicious behavior based on the behavior label.

[0100] As an optional implementation, a suspicious behavior tag library is pre-set, containing multiple suspicious behavior tags. The behavior tag is sequentially compared with each suspicious behavior tag in the library. If a suspicious behavior tag in the library matches the behavior tag, the target is determined to have engaged in suspicious behavior.

[0101] As another optional implementation, the target's past behavior tags and corresponding information such as time, location, and environment can be pre-collected to analyze behavioral patterns and construct a pattern library using suspicious behavior tags. A model can then be used to calculate the suspicion probability of the current behavior tag, referencing the pattern library, to determine whether the target is likely to engage in suspicious behavior. Alternatively, thresholds can be pre-set to distinguish between cases requiring attention and those that can be ignored, allowing for pre-screening of behavior tags and reducing the analysis workload.

[0102] S405. If the third stay duration is greater than the third stay duration threshold, or the distance is less than the distance threshold, or the target performs a target action or wanders around in the key area, or the target may exhibit suspicious behavior, it is determined that the server performs monitoring processing.

[0103] Optionally, once one of the above three judgment conditions is met, it is determined that the server performs monitoring processing.

[0104] As an optional implementation, more conditions can be set to participate in the judgment. For example, the number of targets can be determined based on the image sequence. If the number of targets is greater than a preset number, or if the third dwell time is greater than the third dwell time threshold, or if the distance is less than a distance threshold, or if the target performs a target action or wanders around in the key area, or if the target may exhibit suspicious behavior, then it is determined that the server will perform monitoring processing.

[0105] In this embodiment, if it is determined that the third stay duration is greater than the third stay duration threshold, or if it is determined that the distance is less than the distance threshold, or if it is determined that the target performs a target action or wanders around in a key area, or if it is predicted that the target may exhibit suspicious behavior, it is determined that the server performs monitoring processing, thereby ensuring the accuracy of environmental risk judgment.

[0106] As an optional implementation, the method of determining the target's moving trajectory based on the image sequence in the above step S303 may be: reconstructing the spatial moving path of the target in the image sequence based on a tracking algorithm to obtain the target's moving trajectory.

[0107] Among them, the tracking algorithm, for example, combines the YOLO model and the Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) multi-target tracking algorithm.

[0108] As an optional implementation, the image sequence can be preprocessed, including denoising and enhancement, to improve target clarity, and background modeling can be performed to facilitate subsequent detection of moving targets. Then, the image pixel coordinates of each image in the image sequence are converted to real-world coordinates, thereby establishing a mapping relationship from pixel coordinates to world coordinates. The motion region is then directly extracted using frame difference or background subtraction methods, or the target is detected in real time using models such as the YOLO model or Faster R-CNN, and the bounding box coordinates are output as the motion region. The color histogram of the target within the motion region is extracted to distinguish different targets in the detection area, and the target's position, velocity, and acceleration are calculated to assist in subsequent trajectory association. A smooth three-dimensional motion trajectory is then established based on the position, velocity, and acceleration of each target using its real-world coordinates.

[0109] In this embodiment, the spatial movement path of the target in the image sequence is reconstructed based on the tracking algorithm to obtain the target's movement trajectory, thereby improving the target recognition capability and maintaining the continuity of tracking even if the target is occluded or changes in appearance.

[0110] As an optional implementation, the step of predicting the target's behavior label based on the image sequence in the above step S304 may be: based on a convolutional neural network and a recurrent neural network, the image sequence is behavior-classified to obtain the target's behavior label.

[0111] Specifically, an image sequence can be extracted at a fixed frame rate and the target region can be cropped to focus on behavioral features. The image size and pixel values ​​are then normalized to enhance model generalization. Convolutional neural networks (CNNs) such as ResNet and MobileNet are then used to extract single-frame visual features. Representations of key regions are enhanced by combining dilated convolutions or attention mechanisms. Temporal dynamics are then modeled using a recurrent neural network. Specifically, a multi-frame feature sequence consisting of single-frame visual features is input. A long short-term memory (LSTM) or gated recurrent unit (GRU) is used to process the feature sequence frame by frame, step by step. The hidden state fuses the current frame features with the previous state, outputting a hidden representation containing temporal information. The final hidden state of the CNN or the sequence pooling result is then passed through a fully connected layer and softmax to output behavioral category probabilities. A cross-entropy loss is used to optimize classification accuracy to determine the target's behavioral label.

[0112] In this embodiment, the behavior of the image sequence is classified based on a convolutional neural network and a recurrent neural network to obtain the behavior label of the target. This method can improve the ability to distinguish dynamic behaviors and increase the accuracy of determining behavior labels.

[0113] As an optional implementation, if the third stay duration is not greater than the third stay duration threshold, and the distance is not less than the distance threshold, and the target does not perform target actions or wander around in the key area, and the target does not exhibit suspicious behavior, the SoC is put into sleep mode.

[0114] Specifically, if the third stay time is not greater than the third stay time threshold, and the distance is not less than the distance threshold, and the target does not perform target actions or wander around in the critical area, and the target does not exhibit suspicious behavior, then it means that the current environmental risk of the vehicle is low and the high-power SoC does not need to continue working, so the SoC is put to sleep.

[0115] In this embodiment, if the third stay time is not greater than the third stay time threshold, and the distance is not less than the distance threshold, and the target does not perform target actions or wander around in the key area, and the target does not exhibit suspicious behavior, the SoC is put into sleep mode, thereby reducing power consumption and ensuring battery life.

[0116] Next, refer to Figure 5 The overall flow chart of the vehicle monitoring wake-up method is introduced. Figure 5 This is a flow chart of another vehicle monitoring wake-up method provided in an embodiment of the present application.

[0117] S501, MCU detects that the vehicle is parked and turned off; S502, MCU receives infrared acquisition information sent by the infrared acquisition module; S503: The MCU determines whether the change amplitude of the heat source is greater than the amplitude threshold, and whether the first residence time of the heat source in the detection area is greater than the first residence time threshold. If so, execute S504; if not, execute S502. S504, MCU wakes up the AI ​​module; S505, the AI ​​module obtains the heat source identification result based on the infrared collected information; S506: The MCU determines whether any of the following conditions is met: whether the heat source identification result indicates that a target exists in the detection area and whether the confidence of the heat source identification result is greater than the confidence threshold; whether the second dwell time is greater than the second dwell time threshold; and whether the area of ​​the target is greater than the preset area threshold. If not, execute S502; if so, execute S507. S507, MCU wakes up SoC; S508, the SoC initializes the image acquisition module and sets the frame rate of the image acquisition module to a first frame rate; S509. The SoC determines whether any of the following conditions is met: whether the third stay duration is greater than the third stay duration threshold; whether the distance is less than the distance threshold; whether the target performs a target action or wanders around in the key area; whether the target may exhibit suspicious behavior. If not, execute S510; if so, execute S511. S510: MCU puts the SoC into sleep mode and executes S502. S511, the SoC adjusts the frame rate of the image acquisition module to a second frame rate; S512, SoC activates the communication module; S513. The SoC sends the image sequence acquired by the image acquisition module to the server through the communication module to perform monitoring processing.

[0118] An embodiment of the present application also provides a vehicle monitoring wake-up device for executing the above-mentioned vehicle monitoring wake-up method.

[0119] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A vehicle monitoring wake-up method, characterized in that: The method comprises: The microcontroller unit MCU receives the infrared acquisition information sent by the infrared acquisition module in real time, and determines whether to wake up the artificial intelligence AI module in the MCU according to the infrared acquisition information; If so, the AI ​​module is awakened, and the AI ​​module obtains a heat source identification result according to the infrared collection information, and the MCU determines whether to wake up the system-on-chip SoC according to the heat source identification result of the AI ​​module and the infrared collection information; If so, waking up the SoC, initializing the image acquisition module by the SoC and setting the frame rate of the image acquisition module to the first frame rate, and determining by the SoC whether to execute monitoring processing by the server based on the image sequence acquired by the image acquisition module; If so, the frame rate of the image acquisition module is adjusted to a second frame rate, and the communication module is activated to send the image sequence acquired by the image acquisition module to the server through the communication module to perform monitoring processing, wherein the second frame rate is greater than the first frame rate.

2. The vehicle monitoring wake-up method according to claim 1, characterized in that: The determining whether to wake up the AI ​​module in the MCU according to the infrared collection information includes: determining, based on the infrared collected information, whether a change amplitude of the heat source is greater than an amplitude threshold, and whether a first residence time of the heat source in the detection area is greater than a first residence time threshold; If the change amplitude of the heat source is greater than the amplitude threshold, and the first residence time of the heat source in the detection area is greater than the first residence time threshold, it is determined to wake up the AI ​​module in the MCU.

3. The vehicle monitoring wake-up method according to claim 1, characterized in that: The determining whether to wake up the SoC according to the heat source identification result of the AI ​​module and the infrared collection information includes: determining a second residence time of the heat source in the detection area based on the infrared collected information; Determine whether to wake up the SoC according to the heat source identification result and the second stay time.

4. The vehicle monitoring wake-up method according to claim 3, characterized in that: The determining whether to wake up the SoC according to the heat source identification result and the second stay time includes: If the heat source identification result of the AI ​​module is that there is a target in the detection area and the confidence of the heat source identification result is greater than the confidence threshold, or if the second stay time is greater than the second stay time threshold, or the area of ​​the target identified by the AI ​​module is greater than the preset area threshold, it is determined to wake up the SoC.

5. The vehicle monitoring wake-up method according to claim 1, characterized in that: The determining whether to perform monitoring processing by the server according to the image sequence collected by the image collection module includes: determining a third residence time of the target in the detection area according to the image sequence; determining a distance between a target and a vehicle based on the image sequence; determining a moving trajectory of a target according to the image sequence; predicting a target behavior label based on the image sequence, wherein the behavior label is used to indicate whether the target is suspicious; Determine whether to perform monitoring processing by the server according to the third stay duration, the distance, the movement trajectory, and the behavior tag.

6. The vehicle monitoring wake-up method according to claim 5, characterized in that: The determining whether to perform monitoring processing by the server according to the third stay duration, the distance, the movement trajectory, and the behavior tag includes: Determining whether the third stay duration is greater than a third stay duration threshold; determining whether the distance is less than a distance threshold; Determining, based on the movement trajectory, whether the target is performing a target action or wandering within the key area; Predicting whether the target may exhibit suspicious behavior based on the behavior label; If the third stay duration is greater than the third stay duration threshold, or the distance is less than the distance threshold, or the target performs a target action or wanders around in the key area, or the target may exhibit suspicious behavior, it is determined that the server performs monitoring processing.

7. The vehicle monitoring wake-up method according to claim 5, characterized in that: Determining the moving trajectory of the target according to the image sequence includes: The spatial movement path of the target in the image sequence is reconstructed based on a tracking algorithm to obtain the movement trajectory of the target.

8. The vehicle monitoring wake-up method according to claim 5, characterized in that: Predicting the target's behavior label based on the image sequence includes: Based on convolutional neural networks and recurrent neural networks, behavior classification is performed on the image sequence to obtain behavior labels of the targets.

9. The vehicle monitoring wake-up method according to claim 6, characterized in that: The method further comprises: If the third stay duration is not greater than the third stay duration threshold, and the distance is not less than the distance threshold, and the target does not perform target actions or wander around in the key area, and the target does not exhibit suspicious behavior, the SoC is put to sleep.

10. A vehicle monitoring wake-up device, characterized in that: The vehicle monitoring and waking-up device is used to execute the vehicle monitoring and waking-up method according to any one of claims 1 to 9.

Images