Assisted driving device and system based on video image processing

Abstract

The application relates to the technical field of data processing, in particular to an auxiliary driving device and system based on video image processing, which comprises a frame buffer queue of multi-channel video acquisition module timestamp alignment; a hierarchical calculation task decoupling module identifies the task type in the frame buffer queue and marks a priority label, the priority label comprises a first priority label of a radar data analysis and distance superposition task, a second priority label of a vehicle outline line generation task and a third priority label of a video fusion task, and a data stream with the priority label is output; a calculation module generates a task processing result; a genetic algorithm optimization scheduling module generates a resource allocation strategy and controls the calculation module to execute the task processing; and an output synchronization control module synthesizes a high-priority task result and performs frame rate synchronization output on a low-priority task. The application solves the real-time deterioration problem caused by multi-path high-definition video stream processing through task grading and dynamic resource scheduling, and reduces the image output delay of the auxiliary driving scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an auxiliary driving device and system based on video image processing. BACKGROUND

[0002] The auxiliary driving system based on video image processing relies on the camera equipped on the vehicle to collect the environmental visual information in real time, uses the image processing technology to perform data cleaning and feature extraction, such as identifying key elements through edge detection and object segmentation; then, a machine learning model is integrated to analyze the dynamic scene and predict potential collision or lane deviation risks; by converting the analysis results into real-time warning signals or feedback control, the system directly assists the driver to adjust the operation, effectively reduces the human error, and thus enhances the road traffic safety and driving comfort.

[0003] In the actual application of the existing auxiliary driving based on video image processing, the following technical pain points exist, specifically, the high concurrency processing demand of multiple video streams causes the conflict of computing resource allocation, causing the real-time degradation of key tasks (such as obstacle recognition and collision warning). When the processor needs to perform pixel-level fusion operation of infrared and low-light video, multi-channel high-definition video encoding and decoding, radar data analysis and auxiliary line generation in parallel, resource contention may easily occur in task scheduling; especially in low-light complex road conditions, additional computing resources are consumed for dynamic scene analysis, further intensifying the pipeline blockage. For example, in the night reversing scene, the system needs to process the rear-view video stream, analyze the radar ranging data and superimpose the auxiliary line simultaneously, if the video fusion task occupies excessive computing modules, it will delay the real-time superposition of obstacle distance information, causing the driver to receive delayed warning, increasing the risk of low-speed collision. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides an auxiliary driving device and system based on video image processing, which solves the technical problem that the high computing load caused by the real-time processing demand of multiple high-resolution video streams (such as infrared and low-light fusion) causes the image output delay to exceed the threshold in the auxiliary driving scene, resulting in that the driver cannot respond to the potential collision risk or obstacle information in time.

[0005] To solve the above technical problems, the specific content of the present application is as follows:

[0006] In a first aspect, the present application provides an auxiliary driving device based on video image processing, comprising:

[0007] a multi-channel video acquisition module, which synchronously receives multiple external video signals, performs standardized conversion on the video signals, and outputs a timestamp-aligned frame buffer queue;

[0008] The hierarchical computing task decoupling module receives the frame buffer queue of the multi-channel video acquisition module, identifies the task type of the video data in the frame buffer queue, and marks a priority label according to a preset real-time rule, wherein the priority label comprises: marking a radar data analysis and distance superposition task as a first priority label; marking a vehicle outline generation task as a second priority label; and marking a video fusion task as a third priority label; and outputting the data stream with the priority label.

[0009] The computing module is configured to receive the data stream with the priority label output by the hierarchical computing task decoupling module and the resource allocation instruction sent by the genetic algorithm optimization scheduling module.

[0010] And the computing module is configured to:

[0011] process the data stream corresponding to the first priority label to generate a first priority task processing result;

[0012] process the data stream corresponding to the second priority label to generate a second priority task processing result;

[0013] process the data stream corresponding to the third priority label to generate a third priority task processing result;

[0014] The genetic algorithm optimization scheduling module receives the data stream with the priority label from the hierarchical computing task decoupling module, runs multiple groups of genetic algorithm instances in parallel to generate a resource allocation strategy, the genetic algorithm instance comprises a chromosome code and an adaptability function, the chromosome code represents a processing resource allocation ratio and a task sequence, loads historical scene data to simulate a video processing task flow, evaluates the optimization effect of the resource allocation strategy on task delay, selects the resource allocation strategy with the highest adaptability, and sends a resource allocation instruction generated based on the resource allocation strategy to the computing module, the resource allocation instruction is used to control the computing module to execute task processing according to the resource allocation strategy.

[0015] The output synchronization control module receives the first priority task processing result and the second priority task processing result output by the computing module, and receives the third priority task processing result output by the computing module; the output synchronization control module synthesizes the radar distance analysis data in the first priority task processing result and the vehicle outline data in the second priority task processing result through a hardware superposition layer; performs a frame rate synchronization operation on the third priority task processing result, and outputs a synchronization processing result to a display device.

[0016] Further, the auxiliary driving device based on video image processing of the present application is characterized in that the multi-channel video acquisition module is configured to:

[0017] receive an analog high-definition signal, perform signal conversion processing, and output a digital video stream;

[0018] Receive composite video blanking synchronization infrared signal, implement data unpacking output YUV color coding format data through universal serial bus protocol;

[0019] Receive digital video stream and YUV color coding format data, adopt double buffering mechanism to cache input frame, store next frame into independent storage area when current frame is processed;

[0020] Perform timestamp calibration processing on video stream data in independent storage area, utilize external clock source to calibrate video stream timestamp corresponding to digital video stream and YUV color coding format data to microsecond level precision, form frame cache queue.

[0021] Further, the auxiliary driving device based on video image processing is configured that the hierarchical computing task decoupling module is configured to:

[0022] Receive video data in frame cache queue, parse resolution parameter and signal source type of metadata of video data;

[0023] According to resolution parameter and signal source type, classify video data into radar data stream, rearview video stream and front view fusion video stream;

[0024] Inject first priority label to classified radar data stream, inject second priority label to classified rearview video stream, and inject third priority label to classified front view fusion video stream, to form data stream with priority label.

[0025] Further, the auxiliary driving device based on video image processing is configured that the genetic algorithm optimization scheduling module is configured to:

[0026] Load preset historical scene data set, generate multi-path video concurrent peak load based on historical scene data set;

[0027] Inject data stream with priority label into preset virtualization environment, apply resource allocation strategy to process data stream with priority label;

[0028] Record completion delay of first priority task, second priority task and third priority task under resource allocation strategy;

[0029] Input recorded completion delay into fitness function to calculate evaluation value, variables of fitness function include maximum delay of task and resource utilization rate, and function weight coefficient is dynamically adjusted according to processor load rate and memory occupancy rate.

[0030] Further, the auxiliary driving device based on video image processing is configured that the genetic algorithm optimization scheduling module is further configured to:

[0031] Response to received vehicle gear signal or bus instruction;

[0032] When the vehicle gear signal or the bus instruction represents the reverse mode, an instruction is sent to the computing module to suspend processing of the data stream corresponding to the third priority task, and the released computing resources are reallocated to the first priority task and the second priority task;

[0033] When the vehicle gear signal or the bus instruction represents the forced radar mode, an instruction is sent to the computing module to make the first priority task monopolize the processing resource proportion defined by the resource allocation strategy;

[0034] The resource allocation strategy is a resource allocation strategy whose fitness evaluation value reaches a preset threshold.

[0035] Further, the auxiliary driving device based on video image processing of the present application, the output synchronization control module is configured to:

[0036] Receiving radar distance analysis data in the first priority task processing result, calling a preset video processing engine to write the radar distance analysis data in pixel format to the hardware overlay layer of the display controller;

[0037] Receiving the third priority task processing result, storing the third priority task processing result in the video memory;

[0038] When the vertical synchronization signal of the display device is at the starting point of the vertical blanking period, triggering the physical transmission channel of the direct memory access controller to transmit the third priority task processing result in the video memory;

[0039] Monitoring the actual image output delay of the high-definition multimedia interface, and feeding back the actual image output delay to the fitness function weight adjustment parameter of the genetic algorithm optimization scheduling module.

[0040] Further, the auxiliary driving device based on video image processing of the present application further comprises:

[0041] The hierarchical computing task decoupling module receives the sensor absence signal, and marks the task corresponding to the absent sensor as invalid state;

[0042] The genetic algorithm optimization scheduling module receives the invalid state mark, and updates the weight parameter of the fitness function to shield the task corresponding to the invalid state mark;

[0043] Based on the updated fitness function, the resource allocation strategy is regenerated, an instruction is sent to the computing module to recover the processing resources of the task corresponding to the invalid state mark, and the recovered resources are allocated to the priority task processing flow corresponding to the valid task, the priority task processing flow including the first priority task processing flow, the second priority task processing flow and the third priority task processing flow.

[0044] Further, the auxiliary driving device based on video image processing of the present application, the genetic algorithm optimization scheduling module is further configured to:

[0045] Obtain external sensor data, determine the preset distance range boundary value;

[0046] In the video fusion task processing process, when the input data is within the preset distance range boundary value, a high-precision registration algorithm is executed; when the input data exceeds the preset distance range boundary value, a fast alignment algorithm is executed;

[0047] The light intensity is detected by the photosensitive sensor, and when the light intensity is lower than 50 lux, the resolution of the collected video data is reduced to generate reduced sample frame data, and the reduced sample frame data is input into the third priority task processing flow to generate the third priority task processing result.

[0048] Further, the auxiliary driving device based on video image processing of the present application further comprises a fault tolerance module, the fault tolerance module is connected with the genetic algorithm optimization scheduling module, and the fault tolerance module is configured to:

[0049] Monitor the genetic algorithm instance iteration period of the genetic algorithm optimization scheduling module, and record the iteration period length;

[0050] When the iteration period length exceeds the preset timeout threshold, a hardware reset instruction is sent to the calculation module;

[0051] Load the pre-stored degradation scheduling strategy to replace the current resource allocation strategy, and inject the degradation scheduling strategy into the calculation module for execution.

[0052] Secondly, the auxiliary driving system based on video image processing provided by the present application is applied to the auxiliary driving device based on video image processing, and comprises:

[0053] The multi-channel video acquisition module is configured to synchronously receive multiple external video signals, perform standardized conversion on the video signals, and output a timestamp-aligned frame buffer queue;

[0054] The hierarchical calculation task decoupling module is connected with the multi-channel video acquisition module and is configured to:

[0055] Receive the frame buffer queue, identify the task type of the video data, and mark a priority label according to a preset real-time rule, including: marking the radar data analysis and distance superposition task as a first priority label, marking the vehicle outline generation task as a second priority label, and marking the video fusion task as a third priority label, and outputting a data stream with a priority label;

[0056] The computing module is connected with the hierarchical computing task decoupling module and is configured to: receive a data stream with a priority label and a resource allocation instruction, process a data stream corresponding to a first priority label to generate a first priority task processing result, process a data stream corresponding to a second priority label to generate a second priority task processing result, and process a data stream corresponding to a third priority label to generate a third priority task processing result;

[0057] The genetic algorithm optimization scheduling module is connected with the hierarchical computing task decoupling module and the computing module and is configured to: receive a data stream with a priority label, run multiple groups of genetic algorithm instances in parallel to generate a resource allocation strategy, load historical scene data to simulate a video processing task flow, and send a resource allocation instruction generated based on the resource allocation strategy to the computing module, where the resource allocation instruction is used to control the computing module to execute task processing according to the resource allocation strategy;

[0058] The output synchronization control module is connected with the computing module and is configured to: receive the first priority task processing result, the second priority task processing result and the third priority task processing result output by the computing module, synthesize radar distance analysis data and vehicle outline line data through a hardware superposition layer, perform frame rate synchronization operation on the third priority task processing result, and output a synchronization processing result to a display device.

[0059] The fault tolerance module is connected with the genetic algorithm optimization scheduling module and the computing module and is configured to: monitor an iteration period length of a genetic algorithm instance, and when the iteration period length exceeds a preset timeout threshold, send a hardware reset instruction to the computing module and inject a degraded scheduling strategy to the computing module to replace a current resource allocation strategy.

[0060] The present application has the following advantages:

[0061] The application has the beneficial effects that: the multi-channel video acquisition module eliminates the time sequence difference of multi-source video streams through signal standardization and timestamp alignment operation, and provides a time sequence unified data source for task classification; the classification calculation task decoupling module injects priority labels into tasks based on resolution and signal source characteristics, establishes a task mapping system with radar analysis as the first priority, vehicle outline generation as the second priority, and video fusion as the third priority; the calculation module combines the genetic algorithm to optimize the dynamic resource allocation instruction of the scheduling module, executes differential processing according to the label priority, and guarantees the calculation resources of high-priority tasks; the genetic algorithm optimization scheduling module generates a resource allocation strategy through historical scene simulation and fitness function dynamic evaluation to reduce the maximum delay of tasks; the output synchronization control module uses a hardware overlay layer and a vertical synchronization trigger mechanism to accelerate the output of key information, and feeds back delay data to optimize the scheduling weight; the fault tolerance module switches to a degradation strategy after iteration timeout to maintain the basic functions of the system, and the whole system cooperates through task classification processing, dynamic resource scheduling and hardware acceleration to solve the real-time degradation problem caused by multi-channel high-resolution video stream processing and reduce the image output delay in the auxiliary driving scene. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on the drawings.

[0063] Figure 1 The system architecture diagram of the auxiliary driving system based on video image processing provided by the embodiment of the present application.

[0064] Figure 2 The video display position and scale diagram in the display provided by the embodiment of the present application.

[0065] Figure 3 The radar distance superposition position diagram provided by the embodiment of the present application.

[0066] Figure 4 The system power supply and control architecture provided by the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the technical solutions of the present application clearer, the following will combine the specific embodiments of the present application and the corresponding drawings to clearly and completely describe the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. The following will combine the drawings to specifically describe the present application provided by the embodiments of the present application. In order to better understand the purpose of the present application, the following will further describe the present application in detail.

[0068] In a first aspect, the application provides an auxiliary driving device based on video image processing, comprising:

[0069] A multi-channel video acquisition module synchronously receives multiple external video signals, performs standardized conversion on the video signals, and outputs a frame buffer queue with timestamp alignment;

[0070] A hierarchical computing task decoupling module receives the frame buffer queue of the multi-channel video acquisition module, identifies the task type of the video data in the frame buffer queue, marks a priority tag according to a preset real-time rule, the priority tag includes: marking the radar data analysis and distance superposition task as a first priority tag; marking the vehicle outline generation task as a second priority tag; marking the video fusion task as a third priority tag; and outputting a data stream with a priority tag;

[0071] A computing module configured to receive the data stream with a priority tag output by the hierarchical computing task decoupling module and the resource allocation instruction sent by the genetic algorithm optimization scheduling module;

[0072] Processing the data stream corresponding to the first priority tag generates a first priority task processing result;

[0073] Processing the data stream corresponding to the second priority tag generates a second priority task processing result;

[0074] Processing the data stream corresponding to the third priority tag generates a third priority task processing result;

[0075] A genetic algorithm optimization scheduling module receives the data stream with a priority tag from the hierarchical computing task decoupling module, and performs: running multiple groups of genetic algorithm instances in parallel to generate resource allocation strategies, the genetic algorithm instance includes chromosome coding and fitness function, the chromosome coding represents the processing resource allocation ratio and task sequence; loading historical scene data to simulate video processing task flow, evaluating the optimization effect of the resource allocation strategy on task delay; selecting the resource allocation strategy with the highest fitness, and sending the resource allocation instruction generated based on the resource allocation strategy to the computing module, the resource allocation instruction is used to control the computing module to execute task processing according to the resource allocation strategy;

[0076] An output synchronization control module receives the first priority task processing result and the second priority task processing result output by the computing module, and receives the third priority task processing result output by the computing module; the output synchronization control module synthesizes the radar distance analysis data in the first priority task processing result and the vehicle outline data in the second priority task processing result through a hardware superposition layer; performs frame rate synchronization operation on the third priority task processing result, and outputs the synchronization processing result to a display device.

[0077] The system power supply and control architecture adopted by the embodiment of the present application is described below Figure 4 The power module board provides redundant power supply for the whole system, the power control board receives the instruction of the genetic algorithm optimization scheduling module to switch the power supply path (such as closing the front view unit power supply in the reverse mode), and the video processing board carries the functions of the hierarchical computing task decoupling module, the computing unit and the genetic algorithm optimization scheduling module.

[0078] The multi-channel video acquisition module synchronously receives multiple external video signals, including AHD high-definition video signals and CVBS infrared video signals. The analog high-definition signal is subjected to signal conversion operation to output digital video stream; the CVBS infrared signal is subjected to USB video protocol unpacking operation to output YUV422 format data. A double buffering mechanism is adopted to cache the input frame, and the next frame is stored in an independent storage area while the current frame is processed. The external clock source is used to perform timestamp alignment operation on the video stream data in the independent storage area. The video stream timestamps corresponding to the digital video stream and the YUV422 format data are formed into a timestamp-aligned frame cache queue.

[0079] The hierarchical computing task decoupling module receives the video data in the frame cache queue, parses the metadata of the video data to obtain resolution parameters and signal source types. According to the analysis results of the resolution parameters and the signal source types, the video data is classified into radar data stream, rear view video stream and front view fusion video stream. The first priority label is injected into the classified radar data stream, the second priority label is injected into the classified rear view video stream, and the third priority label is injected into the classified front view fusion video stream, forming a data stream with priority labels and outputting to the computing module.

[0080] The computing module receives the data stream with priority labels output by the hierarchical computing task decoupling module and the resource allocation instruction. The data stream corresponding to the first priority label is processed to generate the first priority task processing result, the data stream corresponding to the second priority label is processed to generate the second priority task processing result, and the data stream corresponding to the third priority label is processed to generate the third priority task processing result. The execution process of the computing module is controlled by the resource allocation instruction, and the processing resources are allocated according to the resource allocation strategy to execute the task processing operation.

[0081] The genetic algorithm optimization scheduling module receives the priority-labeled data stream from the hierarchical computing task decoupling module. Multiple sets of genetic algorithm instances are run in parallel to generate resource allocation strategies, including chromosome encoding and fitness function, which represents the processing resource allocation ratio and task sequence. A preset historical scenario dataset is loaded to generate multiple video concurrent peak loads based on the historical scenario dataset, and the priority-labeled data stream is injected into the simulation environment to process the data stream using the resource allocation strategy. The completion time delay of the first, second, and third priority tasks under the resource allocation strategy is recorded, and the recorded completion time delay is input into the fitness function to calculate the evaluation value. The fitness function variables include the maximum task delay and resource utilization, and the function weight coefficient is dynamically adjusted according to the processor load rate and memory occupancy rate. The resource allocation strategy with the highest fitness is selected, and resource allocation instructions are sent to the computing module.

[0082] The output synchronization control module receives the first, second, and third priority task processing results output by the computing module. The radar range resolution data in the first priority task processing result and the vehicle outline line data in the second priority task processing result are synthesized through a hardware overlay layer. Frame rate synchronization operation is performed on the third priority task processing result, and direct memory access transmission of the third priority task processing result in the video memory is triggered when the vertical synchronization signal is at the starting point of the vertical blanking period. The actual image output delay of the high-definition multimedia interface is monitored, and the actual image output delay is fed back to the fitness function weight adjustment parameter of the genetic algorithm optimization scheduling module.

[0083] The data flow between modules forms a closed loop: the video signal is standardized into a frame buffer queue by the acquisition module; the decoupling module classifies the video data and injects priority labels; the computing module processes the data stream according to the priority labels to generate task results; the genetic algorithm module dynamically optimizes the resource allocation strategy; and the output module synthesizes the results and feeds back the delay parameters. The technical solution solves the real-time degradation problem caused by the computing load of multiple high-resolution video streams through priority task hierarchical processing and genetic algorithm dynamic scheduling.

[0084] Specifically, the auxiliary driving device based on video image processing of the present application, the multi-channel video acquisition module is configured to:

[0085] receive an analog high-definition signal and perform signal conversion processing to output a digital video stream;

[0086] receive a composite video blanking synchronization infrared signal and implement data unpacking through a universal serial bus protocol to output YUV color encoding format data;

[0087] The multi-channel video acquisition module generates a frame buffer queue, including:

[0088] Receive digital video stream and YUV422 format data, use double buffering mechanism to cache input frame, store next frame in independent storage area while current frame is processed;

[0089] Perform timestamp alignment operation on video stream data in independent storage area, use external clock source to form frame cache queue for video stream timestamp corresponding to digital video stream and YUV422 format data.

[0090] Multi-channel video acquisition module receives external input AHD high-definition video signal, converts analog high-definition signal into BT.656 standard digital stream through special video decoding chip. Synchronously receive CVBS infrared video signal, and use USB video protocol to unpack CVBS signal and output YUV422 format data. The conversion process retains the original video resolution and frame rate parameters, and eliminates the influence of signal source difference on subsequent processing.

[0091] When generating the frame cache queue, the module receives digital video stream and YUV422 format data as input sources. Double buffering mechanism is used to manage video frame data: when the current frame enters the processing flow, the next frame video data is stored in the independent storage area in advance. The independent storage area is a special cache space divided by DDR3 memory, and the depth is set to 8 frames to cope with the frame rate fluctuation of the input source. The double buffering mechanism avoids frame loss during data processing.

[0092] Perform timestamp alignment operation on video stream data in independent storage area. Use GPS second pulse signal as external clock source to trigger hardware interrupt to align all input video streams. The calibration process synchronizes nine-way video timestamp to microsecond level precision, eliminating the delay difference of multi-channel signal transmission. The calibrated video data is rearranged in time sequence to form a standardized frame cache queue.

[0093] The frame cache queue output includes two core features: timestamp alignment to microsecond level error range; and video format unified to YUV420 plane format. The standardized output provides consistent data source in time dimension and pixel format for subsequent modules.

[0094] Specifically, the auxiliary driving device based on video image processing of the application is configured as:

[0095] The way in which the hierarchical computing task decoupling module identifies the task type includes:

[0096] Receive video data in the frame cache queue, parse the metadata of the video data to obtain resolution parameters and signal source types;

[0097] According to the parsing results of the resolution parameters and the signal source types, classify the video data into radar data stream, rearview video stream and front-view fusion video stream;

[0098] The first priority label is injected into the classified radar data stream, the second priority label is injected into the classified rear-view video stream, and the third priority label is injected into the classified front-view fusion video stream, thereby forming a data stream with priority labels.

[0099] The hierarchical computing task decoupling module receives a frame buffer queue output by the multi-channel video acquisition module. Metadata of the video data in the frame buffer queue is parsed, and a resolution parameter and a signal source type are extracted. The resolution parameter includes a horizontal pixel number and a vertical pixel number, and the signal source type is identified as an AHD camera, a CVBS infrared sensor, or a radar interface. The metadata parsing is achieved based on decoding of header information of the video stream, and physical characteristics of different data sources are distinguished.

[0100] According to the parsing results of the resolution parameter and the signal source type, the video data is classified into a radar data stream, a rear-view video stream, and a front-view fusion video stream. The classification rule is that when the signal source type is a radar interface and the resolution is lower than a threshold, it is classified into a radar data stream; when the signal source type is an AHD camera and the resolution meets the high-definition standard, it is classified into a rear-view video stream; and when the signal source type is a CVBS infrared sensor and the resolution matches a preset range, it is classified into a front-view fusion video stream. The classification process realizes data label binding through an OpenCL kernel to generate a classification intermediate result.

[0101] The first priority label is injected into the classified radar data stream, and the label code is a binary number 0x01. The second priority label is injected into the classified rear-view video stream, and the label code is 0x02. The third priority label is injected into the classified front-view fusion video stream, and the label code is 0x04. The label injection operation is realized by rewriting a metadata field of the video stream, and a priority identifier is embedded in the header of the data stream.

[0102] The data stream with priority labels includes three types of data entities: the radar data stream carrying the 0x01 label carries distance resolution instructions; the rear-view video stream carrying the 0x02 label includes vehicle contour coordinate information; and the front-view fusion video stream carrying the 0x04 label includes infrared and low-light fusion frame data. The data stream with priority labels is output to a computing module and a genetic algorithm optimization scheduling module for subsequent differentiated processing.

[0103] Specifically, the auxiliary driving device based on video image processing of the application is configured, and the genetic algorithm optimization scheduling module is configured to:

[0104] The process of executing the simulated video processing task flow by the genetic algorithm optimization scheduling module includes:

[0105] A preset historical scene data set is loaded, and a multi-path video concurrent peak load is generated based on the historical scene data set;

[0106] Injecting the priority-labeled data stream into a preset virtualization environment, and processing the priority-labeled data stream by using a resource allocation strategy;

[0107] Recording the completion time delay of the first priority task, the second priority task and the third priority task under the resource allocation strategy;

[0108] Inputting the recorded completion time delay into a fitness function to calculate an evaluation value, the variables of the fitness function including the maximum delay of the task and the resource utilization rate, and the weight coefficients of the function being dynamically adjusted according to the processor load rate and the memory occupancy rate.

[0109] The genetic algorithm optimization scheduling module loads a preset historical scene data set, and the data set includes multi-path video concurrent peak load parameters in a typical driving scene. The historical scene data covers video stream features in low-light reversing, high-speed cruising and complex intersection scenes. The reproduction process simulates nine-path video input concurrent pressure and restores the calculation resource contention state in the actual road conditions.

[0110] Injecting the priority-labeled data stream output by the hierarchical calculation task decoupling module into the simulation environment. The simulation environment is pre-constructed on a virtualization platform and includes a processor core architecture consistent with the physical calculation module. The priority-labeled data stream is processed by using a resource allocation strategy, and the resource allocation strategy includes the processor core ratio and the task scheduling sequence defined by the chromosome coding. The processing process completely reproduces the priority task execution pipeline.

[0111] Recording the completion time delay of the first priority task, the second priority task and the third priority task under the resource allocation strategy. The recording process monitors the task start timestamp and the result output timestamp, and calculates the time difference value as the single task time delay. The maximum time delay, the average time delay and the time delay variance of each priority task in the simulation environment are counted to form a time delay index set.

[0112] Inputting the recorded completion time delay index set into a fitness function to calculate an evaluation value. The fitness function adopts a multi-dimensional weighted calculation model, and the variables of the function include the maximum delay of the task and the resource utilization rate. The weight coefficients of the function are dynamically adjusted according to the real-time system state: when the system detects a high load state, the weight of the maximum delay of the task is increased; when the system is in a low load state, the weight of the resource utilization rate is increased. The evaluation value quantifies the comprehensive performance of the resource allocation strategy.

[0113] Specifically, the auxiliary driving device based on video image processing of the present application, the genetic algorithm optimization scheduling module is further configured to:

[0114] The genetic algorithm optimization scheduling module performs a strategy switching operation in response to an external control signal, including:

[0115] In response to the received vehicle gear signal or bus instruction;

[0116] When the vehicle gear signal or bus instruction represents the reverse mode, an instruction is sent to the computing module to suspend processing of the data stream corresponding to the third priority task, and the released computing resources are reallocated to the first priority task and the second priority task;

[0117] When the vehicle gear signal or bus instruction represents the forced radar mode, an instruction is sent to the computing module to make the first priority task exclusively occupy the processing resource proportion defined by the resource allocation strategy;

[0118] The resource allocation strategy is a resource allocation strategy whose fitness evaluation value reaches a preset threshold.

[0119] The genetic algorithm optimization scheduling module receives a vehicle gear signal or a vehicle-mounted bus instruction through a CAN bus interface. A signal analysis module extracts instruction features: the gear signal includes forward gear, reverse gear and parking gear state codes; the bus instruction includes a forced radar mode trigger identifier. Signal type identification is completed based on a preset protocol library, and a mode determination result is output.

[0120] When the signal analysis result represents the reverse mode, the genetic algorithm optimization scheduling module sends a task suspension instruction to the computing module. The instruction format is a 32-bit control word, the high 16 bits specify the third priority task identifier, and the low 16 bits define the suspension operation code. The computing module stops processing the data stream corresponding to the third priority task after responding to the instruction, and releases the occupied processing core and memory bandwidth. The released computing resources are reallocated to the first priority task and the second priority task processing flow according to the resource allocation strategy.

[0121] When the signal analysis result represents the forced radar mode, the module calls the pre-stored resource allocation strategy with the highest fitness. An exclusive control instruction is sent to the computing module, including the first priority task identifier and the processing resource proportion defined by the resource allocation strategy. The computing module configures a dedicated processing core according to the instruction, prohibits other tasks from occupying the specified proportion of computing resources, and makes the first priority task exclusively occupy the execution environment.

[0122] The resource allocation strategy is derived from the historical evaluation results of the genetic algorithm optimization scheduling module. The resource allocation strategy with the highest fitness is stored in a non-volatile memory, including processor core allocation proportion, memory channel occupancy rate and task execution sequence code. The strategy switching process maintains the integrity of the chromosome code, and realizes that the scheduling logic meets the real-time constraint.

[0123] Specifically, the auxiliary driving device based on video image processing of the application, the output synchronization control module is configured as:

[0124] The hardware overlay layer of the output synchronization control module runs in the following modes:

[0125] receive radar range resolution data in the first priority task processing result, and call a preset video processing engine to write the radar range resolution data in a pixel format into a hardware overlay layer of a display controller;

[0126] receive the third priority task processing result, and store the third priority task processing result into the video memory;

[0127] trigger a physical transmission channel of the direct memory access controller to transmit the third priority task processing result in the video memory when a vertical synchronization signal of the display device is at a starting point of a vertical blanking period;

[0128] monitor an actual image output delay of the HDMI, and feed back the actual image output delay to a fitness function weight adjustment parameter of the genetic algorithm optimization scheduling module.

[0129] The output synchronization control module receives the first priority task processing result output by the calculation module. Radar range resolution data including obstacle coordinates and distance parameters are extracted from the first priority task processing result. A hardware overlay layer interface of a special video processing engine is called to convert the radar range resolution data into an ARGB8888 format pixel block. The converted data is written into a display overlay layer video memory address through a DMA channel. The overlay layer is independent of the main video rendering pipeline.

[0130] The third priority task processing result output by the calculation module is received, and the third priority task processing result is video fusion frame data. The video fusion frame data is stored into a background buffer area of a GPU video memory double buffer area. The video memory area pre-allocates storage space according to resolution and color depth, so as to avoid delay introduced by dynamic memory allocation.

[0131] The display controller monitors the state of the vertical synchronization signal. When the rising edge of the vertical synchronization signal arrives, the direct memory access controller is triggered to start the transmission process. The direct memory access controller reads the video fusion frame data in the background buffer area of the video memory, and transmits the video fusion frame data to the frame buffer area of the HDMI transmitter through a hardware channel. The transmission process skips the CPU intervention, and reduces the data moving delay.

[0132] The actual image output delay of the HDMI is monitored in real time. The delay measurement records the time interval from the video frame entering the video memory to the completion of the output of the HDMI physical port. The actual image output delay value is written into a feedback register of the shared memory. The genetic algorithm optimization scheduling module periodically reads the value of the feedback register, and dynamically adjusts the fitness function weight adjustment parameter. When the weight adjustment parameter increases, the influence proportion of the maximum delay of the task in the resource allocation evaluation is improved.

[0133] Specifically, the auxiliary driving device based on video image processing further comprises:

[0134] When the system detects a sensor missing event:

[0135] The hierarchical computing task decoupling module receives the sensor missing signal, and marks the task corresponding to the missing sensor as an invalid state;

[0136] The genetic algorithm optimization scheduling module receives the invalid state mark, updates the fitness function to shield the weight parameters of the task corresponding to the invalid state mark;

[0137] Based on the updated fitness function, a resource allocation strategy is regenerated, and an instruction is sent to the computing module to recover the processing resources of the task corresponding to the invalid state mark, and the recovered resources are allocated to the priority task processing flow corresponding to the valid task, including the first priority task processing flow, the second priority task processing flow and the third priority task processing flow.

[0138] The system monitors the sensor connection state through a hardware probe. When the sensor power supply is interrupted or the communication timeout lasts more than a preset period, it is determined that a sensor missing event has occurred. The sensor missing signal includes a sensor type identifier and a fault code, and the signal format follows the CAN bus error frame standard.

[0139] The hierarchical computing task decoupling module receives the sensor missing signal. According to the sensor type identifier, a preset mapping table is queried to determine the task type associated with the missing sensor. The task corresponding to the missing sensor is marked as an invalid state, and the invalid state mark is written in the form of a binary flag bit in the data stream metadata header. The marked data stream retains the original data structure but carries an invalid identification.

[0140] The genetic algorithm optimization scheduling module receives the data stream with the invalid state mark. The type identifier of the task corresponding to the invalid state mark is parsed, and the weight parameter configuration of the fitness function is updated. In the fitness function calculation model, the weight parameters of the task corresponding to the invalid state mark are shielded, so that the resource allocation strategy ignores the related task indicators. Based on the updated fitness function, a new genetic algorithm instance is run to generate a new resource allocation strategy.

[0141] The resource recovery instruction is sent to the computing module, and the instruction includes the processing resource identifier of the task corresponding to the invalid state mark. The computing module releases the memory allocation and cache space of the specified processing core according to the instruction. The recovered computing resources are reallocated to the current valid task processing flow, and the resource allocation ratio follows the definition of the newly generated resource allocation strategy.

[0142] Specifically, the auxiliary driving device based on video image processing of the present application, the genetic algorithm optimization scheduling module is further configured to:

[0143] The genetic algorithm optimization scheduling module acquires external sensor data and determines a preset distance range boundary value;

[0144] In the video fusion task processing process, when the input data is within the preset distance range boundary value, a high-precision registration algorithm is executed; when the input data exceeds the preset distance range boundary value, a fast alignment algorithm is executed.

[0145] The ambient light intensity is detected by a photosensitive sensor, and when the light intensity is lower than 50 lux, resolution down-sampling processing is performed on the collected video data to generate down-sampled frame data, and the down-sampled frame data is input into a third priority task processing flow to generate a third priority task processing result.

[0146] The genetic algorithm optimization scheduling module obtains external sensor data through a vehicle-mounted bus, and the sensor data includes distance measurement values of a laser range finder or a millimeter wave radar. Based on the statistical analysis result of the distance measurement values, a preset distance range boundary value is determined, and the boundary value divides the distance threshold of the near-field region and the far-field region. The distance range boundary value is stored in a configuration register for real-time calling.

[0147] In the video fusion task processing process, the module detects the spatial coordinate information of the input data. When the input data coordinates are within the near-field region defined by the preset distance range boundary value, a high-precision registration algorithm is executed. The high-precision registration algorithm uses feature point matching and perspective transformation matrix calculation to realize sub-pixel level alignment of multiple source video streams. When the input data coordinates exceed the far-field region defined by the boundary value, a fast alignment algorithm is executed. The fast alignment algorithm is based on phase correlation and block matching technology, and reduces the computational complexity to meet the real-time constraint.

[0148] For the video data collected in an environment with light intensity lower than 50 lux detected by a photosensitive sensor, a resolution down-sampling operation is performed. The down-sampling operation uses a bilinear interpolation algorithm to convert the original resolution video frame into down-sampled frame data. The down-sampled frame data retains key feature information but reduces the number of pixels, and the output data format matches the input specification of the video fusion task processing flow. The down-sampled frame data is directly input into the processing queue of the video fusion task processing flow through the memory channel.

[0149] Specifically, the auxiliary driving device based on video image processing of the application further comprises a fault tolerance module connected to the genetic algorithm optimization scheduling module, and the fault tolerance module is configured to:

[0150] Monitor the iteration period of the genetic algorithm instance of the genetic algorithm optimization scheduling module, and record the iteration period length;

[0151] When the iteration period length exceeds a preset timeout threshold, a hardware reset instruction is sent to the computing module;

[0152] Load the pre-stored degradation scheduling strategy to replace the current resource allocation strategy, and inject the degradation scheduling strategy into the computing module for execution.

[0153] The fault-tolerant module monitors the iteration period of the genetic algorithm instance of the genetic algorithm optimization scheduling module through a hardware timer. The complete iteration period from the generation of the initial population to the completion of the fitness evaluation of each genetic algorithm instance is recorded. The iteration period data is stored in a circular buffer, retaining the history of the last ten iterations.

[0154] When the iteration period is monitored to exceed the preset timeout threshold, the fault-tolerant module sends a hardware reset instruction to the computing module through the system bus. The hardware reset instruction includes three levels of reset signals: a first level reset to clear the computing module pipeline tasks; a second level reset to recycle memory allocation resources; and a third level reset to reset the processor core state register. The reset operation forcibly terminates the non-real-time tasks being executed.

[0155] Load the pre-stored degraded scheduling strategy in the non-volatile memory. The degraded scheduling strategy is a fixed resource allocation scheme, with a priority order of first priority task, second priority task, and third priority task. The degraded scheduling strategy replaces the current resource allocation strategy by rewriting the strategy register of the genetic algorithm optimization scheduling module. The degraded scheduling strategy is injected into the computing module execution environment through the memory mapping interface, overriding the original strategy configuration parameters.

[0156] In a second aspect, referring to Figure 1 The application provides an auxiliary driving system based on video image processing, which is applied to an auxiliary driving device based on video image processing and includes:

[0157] A multi-channel video acquisition module is configured to synchronously receive multiple external video signals, perform standardized conversion on the video signals, and output a frame buffer queue with timestamp alignment.

[0158] A hierarchical computing task decoupling module is connected to the multi-channel video acquisition module and is configured to:

[0159] receive the frame buffer queue, identify the task type of the video data, and mark a priority label according to a preset real-time rule, including marking the radar data analysis and distance superposition task as a first priority label, marking the vehicle outline generation task as a second priority label, and marking the video fusion task as a third priority label, and output a data stream with the priority label;

[0160] A computing module is connected to the hierarchical computing task decoupling module and is configured to receive the data stream with the priority label and a resource allocation instruction, process the data stream corresponding to the first priority label to generate a first priority task processing result, process the data stream corresponding to the second priority label to generate a second priority task processing result, and process the data stream corresponding to the third priority label to generate a third priority task processing result.

[0161] The genetic algorithm optimization scheduling module is connected with the hierarchical calculation task decoupling module and the calculation module, and is configured to: receive data streams with priority labels, run multiple groups of genetic algorithm instances in parallel to generate resource allocation strategies, load historical scene data to simulate video processing task flows, and send resource allocation ratio instructions generated based on the resource allocation strategies to the calculation module, where the resource allocation ratio instructions are used to control the calculation module to execute task processing according to the resource allocation strategies;

[0162] The output synchronization control module is connected with the calculation module and is configured to: receive the first-priority task processing result, the second-priority task processing result and the third-priority task processing result output by the calculation module, synthesize radar range resolution data and vehicle outline line data through a hardware overlay layer, perform frame rate synchronization on the third-priority task processing result, and output a synchronization processing result to a display device; as shown in Figure 2 The front view fusion image area of the display device is allocated with 1190*720 pixels, the left view area and the right view area each occupy 360*357, and the rear view area is 720*576. The radar range resolution data is written into the specified coordinates of the front view area in the ARGB format through a special video processing engine, and the vehicle outline line data is synthesized into the bottom auxiliary line layer of the rear view area. The multi-picture segmentation ratio and the vertical synchronization signal triggering mechanism cooperate to realize real-time superimposed display of high-priority task processing results. Figure 3 It is further clarified that the rear radar range information is displayed at the bottom central position of the rear view image, the coordinate mapping hardware overlay layer writes the address, and the sub-pixel level rendering of the obstacle distance information is realized.

[0163] The fault tolerance module is connected with the genetic algorithm optimization scheduling module and the calculation module, and is configured to: monitor the iteration period length of the genetic algorithm instance, and when the iteration period length exceeds a preset timeout threshold, send a hardware reset instruction to the calculation module and inject a degraded scheduling strategy to replace the current resource allocation strategy.

[0164] The multi-channel video acquisition module synchronously receives multiple external video signals, including AHD high-definition signals and CVBS infrared signals. The analog high-definition signals are subjected to signal conversion operation to output digital video streams; the CVBS infrared signals are subjected to universal serial bus protocol unpacking operation to output YUV422 format data. A double buffering mechanism is used to cache input frames, and the next frame is stored in an independent storage area while the current frame is being processed. The GPS second pulse signal is used as an external clock source to perform timestamp alignment operation on the video stream data in the independent storage area, to calibrate the timestamps of the nine video signals to the microsecond level, and form a standardized frame buffer queue for output.

[0165] The hierarchical computing task decoupling module receives video data in the frame buffer queue. The video metadata is parsed to obtain resolution parameters and signal source types. The resolution parameters include horizontal pixel number and vertical pixel number. According to the parsing result, the video data is classified: if the signal source is a radar interface and the resolution is lower than a threshold, it is classified as a radar data stream; if the signal source is an AHD camera, it is classified as a rearview video stream; and if the signal source is a CVBS infrared sensor, it is classified as a frontview fusion video stream. The radar data stream is injected with a first priority label 0x01, the rearview video stream is injected with a second priority label 0x02, and the frontview fusion video stream is injected with a third priority label 0x04, forming a data stream with priority labels output to the computing module.

[0166] The computing module receives the data stream with priority labels and resource allocation instructions. The data stream corresponding to the first priority label 0x01 is processed to generate radar distance analysis data; the data stream corresponding to the second priority label 0x02 is processed to generate vehicle outline line data; and the data stream corresponding to the third priority label 0x04 is processed to generate video fusion frame data. The processing process allocates processor cores and memory bandwidth according to the resource allocation instructions, and performs priority task processing operations.

[0167] The genetic algorithm optimization scheduling module receives the data stream with priority labels. Multiple genetic algorithm instances are run in parallel to generate resource allocation strategies, and the chromosome encoding of the genetic algorithm instances includes a CPU / NPU resource allocation proportion bitmap. A historical scene data set is loaded to generate multiple video concurrent peak loads based on the historical scene data set, and the data stream with priority labels is injected into the simulation environment to test the performance of the strategy. Resource allocation instructions are sent to the computing module to control the task processing process, including processor core allocation proportion and task execution sequence encoding.

[0168] The output synchronization control module receives the three types of task processing results output by the computing module. The radar distance analysis data is converted to ARGB8888 format and written to the display overlay layer through the hardware overlay layer; and the video fusion frame data is stored in the video memory background buffer. When the vertical synchronization signal is at the starting point of the vertical blanking period, direct memory access transmission is triggered, and the video memory data is transmitted to the HDMI transmitter. The actual output delay of the high-definition multimedia interface is monitored and fed back to the genetic algorithm module.

[0169] The fault tolerance module monitors the iteration period length of the genetic algorithm instances. When the iteration period length exceeds a preset timeout threshold, a three-level reset instruction is sent to the computing module to clear the processing state. A degraded scheduling strategy is loaded from the non-volatile memory, and the current resource allocation strategy is replaced through a strategy register to inject a fixed priority resource allocation scheme into the computing module execution environment.

[0170] The technical scheme of the present application solves the problem of high-resolution video stream processing delay through a multi-level cooperative mechanism. The multi-channel video acquisition module synchronously receives external video signals, performs standardized conversion, and outputs timestamp-aligned frame buffer queues to eliminate the timing differences of multi-source heterogeneous signals. The hierarchical computing task decoupling module analyzes video metadata and injects priority labels according to the real-time requirements of tasks: the radar data analysis and distance superposition task is marked with the first priority label, the vehicle outline generation task is marked with the second priority label, and the video fusion task is marked with the third priority label. The priority label establishes a mapping relationship between tasks and computing resources.

[0171] The computing module receives data streams with priority labels, combines resource allocation instructions issued by the genetic algorithm optimization scheduling module, and performs differential processing according to label priority. The genetic algorithm optimization scheduling module runs multiple genetic algorithm instances in parallel to generate resource allocation strategies, simulates video processing task streams through historical scene data, and evaluates the optimization effect of the strategy on task delay. The fitness function dynamically adjusts the weight coefficients of the maximum task delay and resource utilization, and selects the optimal strategy to control resource allocation of the computing module.

[0172] The output synchronization control module synthesizes high-priority task processing results through a hardware superposition layer and performs frame rate synchronization operations on low-priority video fusion tasks. The vertical synchronization signal triggers direct memory access to transfer video memory data, and hardware acceleration reduces output delay. The key improvement is that when the system detects the reverse mode, the genetic algorithm module suspends the processing of the third priority task, and the released resources are redistributed to high-priority tasks; when the sensor is missing, the resource strategy is dynamically updated to recover invalid task resources; the video fusion task switches the registration algorithm according to the distance boundary value, and uses downsampling operation in low-illumination environment to reduce the computational load. Task grading and dynamic resource scheduling form a closed loop, ensuring the execution of key tasks and controlling the image output delay within the response threshold.

[0173] The specific implementation steps of the present application are as follows:

[0174] The multi-channel video acquisition module receives eight AHD high-definition video signals and one CVBS infrared video signal through a dedicated interface. The TW9990 decoding chip is used to convert analog high-definition signals into digital video streams, and the USB video protocol is used to unpack the CVBS infrared signal to output YUV422 format data. Double buffering mechanism is used to manage input frames: when the current frame enters the processing flow, the next frame is stored in the independent storage area divided by DDR3 memory. The GPS second pulse triggers a hardware interrupt, and timestamp alignment is performed on the nine video streams, with an accuracy of microseconds, forming a standardized frame buffer queue output. This process eliminates the timing differences of multi-source signals and provides time-consistent data sources for task grading.

[0175] The hierarchical computing task decoupling module parses the metadata of the video data in the frame buffer queue, extracts the resolution parameter and the signal source type. According to a preset classification rule: the resolution is lower than 1280*720 and the signal source is a radar interface, which is classified as a radar data stream; the resolution is greater than or equal to 1920*1080, which is classified as a rear-view video stream; and the rest of the CVBS infrared signal is classified as a front-view fusion video stream. The first priority label 0x01 is injected into the radar data stream, the second priority label 0x02 is injected into the rear-view video stream, and the third priority label 0x04 is injected into the front-view fusion video stream, to generate a data stream with a priority label. The label coding is realized by rewriting the metadata field of the data stream header, and the accurate mapping of the task and the real-time requirement is established.

[0176] The computing module receives the data stream with the priority label and the resource allocation instruction issued by the genetic algorithm optimization scheduling module. The data stream corresponding to the first priority label 0x01 is processed to generate radar distance analysis data; the data stream corresponding to the second priority label 0x02 is processed to generate vehicle outline line coordinates; and the data stream corresponding to the third priority label 0x04 is processed to output a video fusion frame. The resource allocation instruction controls the computing module to allocate processing cores according to the strategy, for example, in the reversing mode, the resources are concentrated to the first two types of tasks. The task processing result retains the original data structure, and the priority identifier is carried when output.

[0177] The genetic algorithm optimization scheduling module loads the historical scene data set to simulate the night reversing working condition, and reproduces the nine-way video 25fps concurrent load. The data stream with the priority label is injected into the virtualization test environment, and the resource allocation strategy defined by the chromosome coding is applied to process the data. The delay of each priority task is recorded: the average delay of the radar analysis task, the peak delay of the vehicle outline line generation, and the delay variance of the video fusion task. The fitness function takes the maximum delay of the task and the resource utilization rate as variables, and the weight coefficient is dynamically adjusted according to the system load. The strategy with the highest fitness is selected and sent to the computing module through the CAN bus, and the response time of the strategy switching is less than 5ms.

[0178] The output synchronization control module calls the IVE engine of Hi3559A, writes the radar distance analysis data into the HDMI overlay layer video memory address in the ARGB8888 format. The video fusion frame is stored in the GPU video memory double-buffer area, and the vertical synchronization signal rising edge triggers the DMA transmission to the HDMI transmitter. The delay from the frame data into the video memory to the HDMI physical output is monitored in real time, and the feedback value is written into the shared register of the genetic algorithm module to drive the dynamic optimization of the fitness function weight.

[0179] The fault-tolerant module monitors the iteration period of the genetic algorithm, and sends a three-level reset instruction to the calculation module after timeout: the first level clears the pipeline task, the second level recycles the memory resource, and the third level resets the core register.

[0180] In the night reversing scene verification, the system responds to the external gear signal switching to the reversing mode. The genetic algorithm module suspends the third priority task processing, and distributes the released NPU resource to the radar analysis and vehicle outline generation task. When the side radar fails, the decoupling module marks the associated task invalid, the scheduling module updates the function to shield the invalid task weight, and the calculation module recycles the resource and redistributes it to the valid task. The video fusion task adopts SIFT feature matching high-precision registration within a distance of 10 meters, and switches to fast phase alignment beyond the range, and low-illumination video is down-sampled to 640x480 for processing. After hardware acceleration output, the obstacle warning delay is compressed to within 50ms, solving the problem of video fusion blocking key warning

[0181] Embodiment one of the present application;

[0182] In the night reversing scene, the multi-channel video acquisition module receives eight AHD high-definition rearview video signals and one CVBS infrared video signal. A special decoding chip converts the analog high-definition signal into a digital video stream, and a universal serial bus protocol unpacks the CVBS signal to output YUV422 data. Double buffering mechanism is adopted to cache input frames, and GPS second pulse triggers hardware interrupt to align nine video time stamps to microsecond level accuracy, forming a standardized frame buffer queue. The hierarchical calculation task decoupling module analyzes the video metadata: radar interface data with resolution lower than 720p is classified as a radar data stream and labeled with a first priority label; analog high-definition signals with 1920x1080 are classified as rearview video streams and labeled with a second priority label; and the remaining infrared signals are classified as front view fusion video streams and labeled with a third priority label.

[0183] The genetic algorithm optimization scheduling module detects the reversing gear signal, sends an instruction to the calculation module to suspend processing of the third priority label data stream, and releases the NPU resource. The resource is redistributed to the radar analysis task of the first priority label and the vehicle outline generation task of the second priority label. The calculation module processes the radar data stream to generate obstacle distance information according to the resource allocation instruction, and processes the rearview video stream to generate real-time auxiliary lines. The output synchronization control module directly writes the radar distance data into the display layer in ARGB format through the Hi3559A hardware overlay layer, and triggers the DMA transmission of the video fusion frame through the vertical synchronization signal. The system completes the superposition of obstacle warning information within 50ms, solving the low-speed collision risk caused by the video fusion task blocking.

[0184] Embodiment two of the present application;

[0185] In the urban peak congestion scenario, the genetic algorithm optimization scheduling module loads the historical data set to simulate nine video 25fps concurrent load. The priority tagged data stream is injected into the virtual environment to test the resource allocation strategy: the chromosome coding defines the resource allocation ratio of CPU 60%, NPU 30%, and GPU 10%. The maximum delay of the first priority task, the average delay of the second priority task, and the delay variance of the third priority task are recorded. The fitness function dynamically adjusts the weight according to the real-time load: the maximum delay weight of the task is increased to 0.7 in high load, and the resource utilization weight is increased to 0.6 in low load. The optimal strategy is selected to control the calculation module to execute task processing.

[0186] When the millimeter wave radar detects that the distance of the preceding vehicle is less than 10 meters, the video fusion task is switched to the high-precision SIFT registration algorithm; when the distance is greater than 30 meters, the fast phase alignment is enabled. In low-illumination environment, the resolution of the CVBS infrared video is reduced to 640x480 to generate a down-sampled frame. When the genetic algorithm iteration period is detected to be overdue, the fault tolerance module immediately sends a three-level reset instruction to the calculation module: clears the pipeline task, recycles memory resources, and resets the core register. Load the degraded scheduling strategy in the ROM, and allocate resources in a fixed proportion (radar analysis 45%, auxiliary line generation 35%, and video fusion 20%). Through dynamic resource scheduling and algorithm optimization, the image output delay is stably controlled within the 100ms threshold under multi-task concurrency, eliminating the complex road condition warning lag problem.

[0187] The technical features of the present application are explained as follows:

[0188] The multi-channel video acquisition module realizes the physical layer integration of multi-source video signals, converts different video streams into standard digital formats through a dedicated decoding chip, and eliminates signal source differences; the double-buffering mechanism cooperates with the external clock source to realize the microsecond-level timestamp alignment of multiple video streams, solving the multi-sensor data timing misalignment problem. The hierarchical computing task decoupling module classifies data streams according to resolution parameters and signal source types, binds radar data streams, rear-view video streams, and front-view fusion video streams with differentiated priority tags based on metadata analysis results, and constructs a real-time hierarchical task system.

[0189] The calculation module serves as a task execution carrier, processes corresponding data streams according to priority tags to generate task results, and dynamically controls the allocation proportion of the processing core through resource allocation instructions to ensure the supply of high-priority task computing resources. The genetic algorithm optimization scheduling module generates chromosome coding through parallel instances, the coding structure maps the processor resource allocation ratio and the task sequence; historical scenario data simulates the actual load pressure, the fitness function adjusts the task maximum delay and resource utilization weight coefficient according to the real-time system state, and realizes dynamic optimization of the resource allocation strategy.

[0190] The output synchronization control module directly synthesizes radar range resolution data and vehicle profile line data by using a hardware superposition layer to avoid software rendering delay; a vertical synchronization signal triggers a direct memory access transmission mechanism to skip CPU intervention, thereby compressing the video fusion result output delay; and output delay feedback forms a closed-loop optimization link. The fault tolerance module identifies algorithm stalemate through iterative cycle monitoring, and a three-level hardware reset instruction clears the abnormal state of the calculation module, and a pre-stored degradation scheduling strategy maintains basic function operation.

[0191] The sensor absence event triggers a task invalidation marking mechanism, the genetic algorithm module updates the fitness function to shield the invalid task weight, and the calculation module recycles processing resources and reallocates them to valid tasks. The video fusion task switches between high-precision registration and fast alignment algorithms based on distance range boundary values, and low-illumination environments use resolution down-sampling operations to balance the calculation load and processing accuracy. The technical solution eliminates real-time degradation caused by high-concurrency video processing through hierarchical task scheduling, dynamic resource optimization, and hardware acceleration collaboration.

[0192] In the present application, AHD (Analog High Definition) refers to an analog high-definition signal, which is an analog standard for transmitting high-definition video through a coaxial cable, supporting resolutions up to 720p or 1080i, used for collecting real-time environmental video from vehicle exterior cameras, and converted into digital signals by a dedicated decoding chip to eliminate transmission delay and noise interference.

[0193] BT.656 is a digital video interface standard developed by the International Telecommunication Union (ITU-R), defining a parallel data transmission protocol including vertical synchronization (VSYNC), horizontal synchronization (HSYNC), and pixel clock signals. In the present application, it is used as the output format after AHD signal conversion, ensuring that the video stream is transmitted in YCbCr 4:2:2 format with a stable frame rate of 25-30fps, providing a time-unified digital source for subsequent processing.

[0194] CVBS (Composite Video Blanking Sync) represents a composite video blanking synchronization signal, which combines luminance, chrominance, blanking pulses, and synchronization signals, and is transmitted through a single-channel coaxial cable. In the present application, it specifically refers to low-illumination environmental video collected by an infrared camera, which is restored to original imaging data after being unpacked by USB protocol, supports NTSC / PAL formats, and has a typical horizontal resolution of 640x480.

[0195] USB (Universal Serial Bus) is a protocol that defines the electrical specification and communication architecture for data transmission between devices, including control transfer, bulk transfer and isochronous transfer modes, in this invention used for implementing protocol unpacking for CVBS infrared signals, realizing zero-copy data transfer with a bandwidth of up to 5Gbps (USB 3.0) and reducing video stream processing delay.

[0196] YUV422 is a color space encoding format in which the luminance component (Y) is fully sampled and the chrominance components (U and V) are half-sampled horizontally (4:2:2 ratio), each pixel occupying 16 bits of storage. In this invention, it is used as the output format after infrared video unpacking, balancing data compression and visual fidelity, reducing memory usage by 30% with chrominance sampling, while preserving key edge features.

[0197] DMA (Direct Memory Access) refers to a direct memory access technology that enables direct data transmission between memory and peripherals through a dedicated hardware controller, bypassing CPU intervention. In this invention, it is used for the physical transmission channel of video frames, with an AXI bus architecture transmission bandwidth of up to 4GB / s and a delay of less than 1μs, significantly reducing the output delay from video memory to HDMI interface.

[0198] HDMI (High-Definition Multimedia Interface) is a high-definition multimedia interface that supports uncompressed audio and video stream transmission, using TMDS encoding protocol, with a maximum bandwidth of 18Gbps (HDMI 2.0). In this invention, it is used to connect display devices to output fusion video, compatible with 1080p@60Hz resolution, and realize control synchronization between devices through CEC protocol.

[0199] Lux is a unit of illumination intensity, 1 lux is equal to 1 lumen per square meter. In this invention, the environmental light threshold (50 lux) is quantified, based on real-time detection by photosensitive sensors, triggering video down-sampling processing. 50 lux corresponds to dusk or low light environment, and below this value, the low-light optimization mode is started.

Claims

1. A driver assistance device based on video image processing, characterized in that, include: The multi-channel video acquisition module simultaneously receives multiple external video signals, performs standardized conversion on the video signals, and outputs a timestamp-aligned frame buffer queue. The hierarchical computing task decoupling module receives the frame buffer queue from the multi-channel video acquisition module, identifies the task type of the video data in the frame buffer queue, and marks the tasks with priority tags according to preset real-time rules. The priority tags include: marking the radar data parsing and distance overlay task as the first priority tag; marking the vehicle outline generation task as the second priority tag; marking the video fusion task as the third priority tag; and outputting the data stream with priority tags. The computing module is configured to receive a data stream with priority tags output by the hierarchical computing task decoupling module, as well as resource allocation instructions sent by the genetic algorithm optimization scheduling module. And configured as follows: Process the data stream corresponding to the first priority label to generate the first priority task processing result; Process the data stream corresponding to the second priority label to generate the second priority task processing result; Process the data stream corresponding to the third priority label to generate the third priority task processing result; The genetic algorithm optimization scheduling module receives a priority-labeled data stream from the hierarchical computation task decoupling module, runs multiple sets of genetic algorithm instances in parallel to generate resource allocation strategies, and the genetic algorithm instances include chromosome encoding and fitness functions. The chromosome encoding represents the processing resource allocation ratio and task sequence. It loads historical scene data to simulate video processing task flow and evaluates the optimization effect of resource allocation strategies on task latency. It selects the resource allocation strategy with the highest fitness and sends a resource allocation ratio instruction generated based on the resource allocation strategy to the computation module. The resource allocation ratio instruction is used to control the computation module to execute task processing according to the resource allocation strategy. The output synchronization control module receives the first priority task processing results and the second priority task processing results output by the calculation module, and receives the third priority task processing results output by the calculation module; the output synchronization control module synthesizes the radar range resolution data in the first priority task processing results and the vehicle outline data in the second priority task processing results through a hardware overlay layer; performs frame rate synchronization operation on the third priority task processing results, and outputs the synchronization processing results to the display device.

2. The driver assistance device based on video image processing according to claim 1, characterized in that, The multi-channel video acquisition module is configured as follows: It receives analog high-definition signals, performs signal conversion processing, and outputs a digital video stream. Receive composite video blanking synchronization infrared signal, and output YUV color encoded format data by performing data unpacking through universal serial bus protocol; It receives digital video streams and YUV color encoding format data, and uses a double buffering mechanism to cache input frames. While processing the current frame, the next frame is stored in an independent storage area. The video stream data in the independent storage area is subjected to timestamp calibration. An external clock source is used to calibrate the timestamps of the digital video stream and the video stream corresponding to the YUV color encoding format data to the microsecond level, forming a frame buffer queue.

3. The video image processing-based driver assistance device according to claim 2, characterized in that, The hierarchical computing task decoupling module is configured as follows: Receive video data from the frame buffer queue, parse the metadata of the video data to obtain resolution parameters and signal source type; Based on the analysis results of resolution parameters and signal source type, the video data is classified into radar data stream, rear-view video stream, and forward-view fusion video stream; A first priority label is injected into the classified radar data stream, a second priority label is injected into the classified rear-view video stream, and a third priority label is injected into the classified forward-view fusion video stream, forming a data stream with priority labels.

4. The video image processing-based driver assistance device according to claim 3, characterized in that, The genetic algorithm optimization scheduling module is configured as follows: Load a preset historical scene dataset and generate peak concurrent load for multiple video streams based on the historical scene dataset; Inject priority-labeled data streams into a preset virtualization environment and apply resource allocation strategies to process the priority-labeled data streams; Record the completion delay of the first, second, and third priority tasks under the resource allocation strategy; The recorded completion latency is input into the fitness function to calculate the evaluation value. The fitness function variables include the maximum task latency and resource utilization. The function weight coefficients are dynamically adjusted according to the processor load rate and memory usage rate.

5. The video image processing-based driver assistance device according to claim 4, characterized in that, The genetic algorithm optimization scheduling module is also configured as follows: In response to received vehicle gear position signal or bus command; When the vehicle gear signal or bus command indicates the reversing mode, a command is sent to the computing module to suspend the processing of the data stream corresponding to the third priority task, and the released computing resources are reallocated to the first priority task and the second priority task. When the vehicle gear signal or bus command indicates a forced radar mode, a command is sent to the computing module to make the first priority task exclusively occupy the proportion of processing resources defined by the resource allocation strategy. The resource allocation strategy is a resource allocation strategy that is based on the fitness evaluation value reaching a preset threshold.

6. The video image processing-based driver assistance device according to claim 5, characterized in that, The output synchronization control module is configured as follows: Receive radar range resolution data from the first priority task processing result, and call the preset video processing engine to write the radar range resolution data into the hardware overlay layer of the display controller in pixel format; Receive the processing result of the third priority task and store the processing result of the third priority task in the display memory. When the vertical synchronization signal of the display device is at the start of the vertical blanking period, the physical transmission channel of the direct memory access controller is triggered to transmit the third priority task processing result in the video memory. Monitor the actual image output delay of the high-definition multimedia interface and feed the actual image output delay back to the fitness function weight adjustment parameters of the genetic algorithm optimization scheduling module.

7. The video image processing-based driver assistance device according to claim 6, characterized in that, Also includes: The hierarchical computing task decoupling module receives sensor missing signals and marks the task corresponding to the missing sensor as invalid. The genetic algorithm optimization scheduling module receives invalid state flags and updates the weight parameters of the task corresponding to the invalid state flag in the fitness function. Based on the updated fitness function, a resource allocation strategy is regenerated, and instructions are sent to the computing module to reclaim the processing resources of tasks corresponding to invalid state markers. The reclaimed resources are then allocated to the priority task processing flow corresponding to the valid tasks. The priority task processing flow includes a first priority task processing flow, a second priority task processing flow, and a third priority task processing flow.

8. The video image processing-based driver assistance device according to claim 7, characterized in that, The genetic algorithm optimization scheduling module is also configured as follows: Acquire data from external sensors and determine the boundary values ​​of a preset distance range; During the video fusion task processing, when the input data is within the preset distance range boundary value, a high-precision registration algorithm is executed; when the input data exceeds the preset distance range boundary value, a fast alignment algorithm is executed. The ambient light intensity is detected by a photosensitive sensor. When the light intensity is below 50 lux, the acquired video data is downsampled to generate downsampled frame data. The downsampled frame data is then input into the third priority task processing flow to generate the third priority task processing result.

9. The video image processing-based driver assistance device according to claim 8, characterized in that, It also includes a fault-tolerant module, which is connected to the genetic algorithm optimization scheduling module. The fault-tolerant module is configured as follows: Monitor the iteration cycle of genetic algorithm instances in the genetic algorithm optimization scheduling module and record the iteration cycle duration; When the iteration cycle duration exceeds the preset timeout threshold, a hardware reset command is sent to the computing module. Load the pre-stored degraded scheduling policy to replace the current resource allocation policy, and inject the degraded scheduling policy into the computing module for execution.

10. A video image processing-based driver assistance system, applied to the video image processing-based driver assistance device as described in any one of claims 1 to 9, characterized in that, include: The multi-channel video acquisition module is configured to simultaneously receive multiple external video signals, perform standardized conversion on the video signals, and output a timestamp-aligned frame buffer queue. The hierarchical computing task decoupling module, connected to the multi-channel video acquisition module, is configured as follows: The system receives a frame buffer queue, identifies the task type of the video data, and marks priority tags according to preset real-time rules, including: marking the radar data parsing and distance overlay task as the first priority tag, marking the vehicle outline generation task as the second priority tag, marking the video fusion task as the third priority tag, and outputting a data stream with priority tags. The computing module, connected to the hierarchical computing task decoupling module, is configured to: receive data streams with priority labels and resource allocation instructions; process the data streams corresponding to the first priority label to generate the first priority task processing results; process the data streams corresponding to the second priority label to generate the second priority task processing results; and process the data streams corresponding to the third priority label to generate the third priority task processing results. The genetic algorithm optimization scheduling module connects the hierarchical computing task decoupling module and the computing module. It is configured to: receive data streams with priority labels, run multiple sets of genetic algorithm instances in parallel to generate resource allocation strategies, load historical scene data to simulate video processing task streams, and send resource allocation instructions generated based on the resource allocation strategies to the computing module. The resource allocation instructions are used to control the computing module to execute task processing according to the resource allocation strategies. The output synchronization control module is connected to the computing module and is configured to: receive the first priority task processing result, the second priority task processing result and the third priority task processing result output by the computing module; synthesize radar range resolution data and vehicle profile data through a hardware overlay layer; perform frame rate synchronization operation on the third priority task processing result; and output the synchronization processing result to the display device. The fault-tolerant module connects the genetic algorithm optimization scheduling module and the computing module. It is configured to monitor the iteration cycle duration of genetic algorithm instances. When the iteration cycle duration exceeds the preset timeout threshold, it sends a hardware reset command to the computing module and injects a degraded scheduling strategy to replace the current resource allocation strategy.

Images