Auxiliary driving device and system based on video image processing

Abstract

The invention relates to the technical field of data processing, in particular to an auxiliary driving device and system based on video image processing, and the device comprises a frame buffer queue with a multi-channel video collection module aligned with timestamps; a hierarchical calculation task decoupling module identifies task types in the frame buffer queue and marks priority labels, including a first priority label of a radar data analysis and distance superposition task, a second priority label of a vehicle profile generation task and a third priority label of a video fusion task, and outputs a data stream with priority labels; the calculation module generates a task processing result; the genetic algorithm optimization scheduling module generates a resource allocation strategy and controls the calculation module to execute task processing; and the output synchronous control module synthesizes a high-priority task result and performs frame rate synchronous output on a low-priority task. According to the method, the problem of real-time degradation caused by multi-path high-definition video stream processing is solved through task grading and dynamic resource scheduling, and the image output delay of the auxiliary driving scene is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an auxiliary driving device and system based on video image processing. BACKGROUND

[0002] The auxiliary driving system based on video image processing relies on the camera equipped on the vehicle to collect the environmental visual information in real time, uses the image processing technology to perform data cleaning and feature extraction, such as identifying key elements through edge detection and object segmentation; then, a machine learning model is integrated to analyze the dynamic scene and predict potential collision or lane deviation risks; by converting the analysis results into real-time warning signals or feedback control, the system directly assists the driver to adjust the operation, effectively reduces the human error, and thus enhances the road traffic safety and driving comfort.

[0003] In the actual application of the existing auxiliary driving based on video image processing, the following technical pain points exist, specifically, the high concurrency processing demand of multiple video streams causes the conflict of computing resource allocation, causing the real-time degradation of key tasks (such as obstacle recognition and collision warning). When the processor needs to perform pixel-level fusion operation of infrared and micro-light video, multi-channel high-definition video encoding and decoding, radar data analysis and auxiliary line generation in parallel, resource contention may easily occur in task scheduling; especially in low-light complex road conditions, additional computing resources are consumed for dynamic scene analysis, further intensifying the pipeline blocking. For example, in the night reversing scene, the system needs to process the rear-view video stream, analyze the radar ranging data and superimpose the auxiliary line simultaneously, if the video fusion task occupies excessive computing modules, it will delay the real-time superposition of obstacle distance information, causing the driver to receive delayed warning, increasing the risk of low-speed collision. SUMMARY

[0004] In view of the deficiencies in the prior art, the present application provides an auxiliary driving device and system based on video image processing, which solves the technical problem that the high computing load caused by the real-time processing demand of multiple high-resolution video streams (such as infrared and micro-light fusion) causes the image output delay to exceed the threshold in the auxiliary driving scene, resulting in that the driver cannot respond to the potential collision risk or obstacle information in time.

[0005] To solve the above technical problems, the specific content of the present application is as follows: In a first aspect, the present application provides an auxiliary driving device based on video image processing, comprising: a multi-channel video acquisition module, which synchronously receives multiple external video signals, performs standardized conversion on the video signals, and outputs a timestamp-aligned frame buffer queue; The hierarchical computing task decoupling module receives the frame buffer queue of the multi-channel video acquisition module, identifies the task type of the video data in the frame buffer queue, and marks the tasks with priority tags according to preset real-time rules. The priority tags include: marking the radar data parsing and distance superposition tasks as the first priority tag; marking the vehicle outline generation task as the second priority tag; marking the video fusion task as the third priority tag; and outputting the data stream with priority tags; a computing module configured to receive the data stream with priority tags output by the hierarchical computing task decoupling module and the resource allocation instruction sent by the genetic algorithm optimization scheduling module; And configured as: Processing the data stream corresponding to the first priority tag to generate a first priority task processing result; Processing the data stream corresponding to the second priority tag to generate a second priority task processing result; Processing the data stream corresponding to the third priority tag to generate a third priority task processing result; The genetic algorithm optimization scheduling module receives the priority-labeled data stream from the hierarchical computing task decoupling module and runs multiple sets of genetic algorithm instances in parallel to generate resource allocation strategies. The genetic algorithm instances include chromosome codes and fitness functions. The chromosome codes represent the processing resource allocation ratio and task sequence. The module loads historical scene data to simulate the video processing task flow and evaluates the optimization effect of the resource allocation strategy on task delay. The module selects the resource allocation strategy with the highest fitness and sends the resource allocation instructions generated based on the resource allocation strategy to the computing module. The resource allocation instructions are used to control the computing module to execute task processing according to the resource allocation strategy. The output synchronization control module receives the first priority task processing result and the second priority task processing result output by the calculation module, and receives the third priority task processing result output by the calculation module; the output synchronization control module synthesizes the radar distance resolution data in the first priority task processing result and the vehicle outline data in the second priority task processing result through the hardware overlay layer; performs a frame rate synchronization operation on the third priority task processing result, and outputs the synchronization processing result to the display device.

[0006] Furthermore, in the driving assistance device based on video image processing of the present invention, the multi-channel video acquisition module is configured as follows: Receive analog high-definition signals, perform signal conversion processing and output digital video stream; Receive composite video blanking synchronization infrared signal, implement data depacketization and output YUV color coding format data through universal serial bus protocol; Receive digital video stream and YUV color encoding format data, use double buffering mechanism to cache input frames, and store the next frame in an independent storage area while the current frame is being processed; The timestamp calibration processing is performed on the video stream data in the independent storage area, and the video stream timestamp corresponding to the digital video stream and YUV color coding format data is calibrated to microsecond level precision by using an external clock source to form a frame buffer queue.

[0007] Further, the auxiliary driving device based on video image processing is configured. The video data in the frame buffer queue is received, and the resolution parameters and signal source types of the video data are obtained by analyzing the metadata of the video data. According to the analysis result of the resolution parameters and the signal source types, the video data is classified into radar data stream, rear-view video stream and front-view fusion video stream. The first priority label is injected into the classified radar data stream, the second priority label is injected into the classified rear-view video stream, and the third priority label is injected into the classified front-view fusion video stream to form a data stream with priority labels.

[0008] Further, the auxiliary driving device based on video image processing is configured. The preset historical scene data set is loaded, and the multi-path video concurrent peak load is generated based on the historical scene data set. The data stream with priority labels is injected into the preset virtualization environment, and the resource allocation strategy is applied to process the data stream with priority labels. The completion delay of the first priority task, the second priority task and the third priority task under the resource allocation strategy is recorded. The recorded completion delay is input into the fitness function to calculate the evaluation value, the fitness function variables include the maximum delay of the task and the resource utilization rate, and the function weight coefficient is dynamically adjusted according to the processor load rate and the memory occupancy rate.

[0009] Further, the auxiliary driving device based on video image processing is configured. In response to the received vehicle gear signal or bus instruction. When the vehicle gear signal or bus instruction represents the reverse mode, an instruction is sent to the calculation module to pause processing the data stream corresponding to the third priority task, and the released calculation resources are reallocated to the first priority task and the second priority task. When the vehicle gear signal or bus instruction represents the forced radar mode, an instruction is sent to the calculation module to make the first priority task exclusively occupy the processing resource proportion defined by the resource allocation strategy. The resource allocation strategy is the resource allocation strategy whose fitness evaluation value reaches the preset threshold.

[0010] Further, the auxiliary driving device based on video image processing is configured. Receive radar distance resolution data from the first priority task processing result, call a preset video processing engine to write the radar distance resolution data into a hardware overlay layer of a display controller in pixel format; receiving a processing result of a third priority task, and storing the processing result of the third priority task in a video memory; When the vertical synchronization signal of the display device is at the starting point of the vertical blanking period, triggering the physical transmission channel of the direct memory access controller to transmit the processing result of the third priority task in the video memory; The actual image output delay of the high-definition multimedia interface is monitored and fed back to the fitness function weight adjustment parameter of the genetic algorithm optimization scheduling module.

[0011] Furthermore, the driving assistance device based on video image processing of the present invention further includes: The hierarchical computing task decoupling module receives the sensor missing signal and marks the task corresponding to the missing sensor as invalid; The genetic algorithm optimization scheduling module receives the invalid state mark and updates the fitness function to shield the weight parameters of the tasks corresponding to the invalid state mark; The resource allocation strategy is regenerated based on the updated fitness function, and instructions are sent to the computing module to reclaim the processing resources of the tasks corresponding to the invalid status mark, and the recovered resources are allocated to the priority task processing flow corresponding to the valid tasks. The priority task processing flow includes the first priority task processing flow, the second priority task processing flow and the third priority task processing flow.

[0012] Furthermore, in the video image processing-based driving assistance device of the present invention, the genetic algorithm optimization scheduling module is further configured to: Obtain external sensor data to determine the boundary value of the preset distance range; During the video fusion task processing, when the input data is within the preset distance range boundary value, the high-precision registration algorithm is executed; when the input data exceeds the preset distance range boundary value, the fast alignment algorithm is executed; The ambient light intensity is detected by a photosensor. When the light intensity is lower than 50 lux, the collected video data is downsampled to generate downsampled frame data, and the downsampled frame data is input into the third priority task processing flow to generate the third priority task processing result.

[0013] Furthermore, the video image processing-based driving assistance device of the present invention further includes a fault tolerance module, which is connected to the genetic algorithm optimization scheduling module and is configured as follows: Monitor the iteration cycle of the genetic algorithm instance of the genetic algorithm optimization scheduling module and record the duration of the iteration cycle; When the iteration cycle length exceeds the preset timeout threshold, a hardware reset instruction is sent to the computing module; Load the pre-stored degraded scheduling strategy to replace the current resource allocation strategy, and inject the degraded scheduling strategy into the computing module for execution.

[0014] In a second aspect, the application provides an auxiliary driving system based on video image processing, which is applied to an auxiliary driving device based on video image processing and includes: A multi-channel video acquisition module configured to synchronously receive multiple external video signals, perform standardized conversion on the video signals, and output a timestamp-aligned frame buffer queue; A hierarchical computing task decoupling module connected to the multi-channel video acquisition module and configured to: Receive the frame buffer queue, identify the task type of the video data, mark a priority label according to a preset real-time rule, including marking the radar data analysis and distance superposition task as a first priority label, marking the vehicle outline generation task as a second priority label, and marking the video fusion task as a third priority label, and output a data stream with the priority label; A computing module connected to the hierarchical computing task decoupling module and configured to: receive the data stream with the priority label and a resource allocation instruction, process the data stream corresponding to the first priority label to generate a first priority task processing result, process the data stream corresponding to the second priority label to generate a second priority task processing result, and process the data stream corresponding to the third priority label to generate a third priority task processing result; A genetic algorithm optimization scheduling module connected to the hierarchical computing task decoupling module and the computing module and configured to: receive the data stream with the priority label, run multiple groups of genetic algorithm instances in parallel to generate a resource allocation strategy, load historical scene data to simulate a video processing task flow, send a resource allocation instruction generated based on the resource allocation strategy to the computing module, and the resource allocation instruction is used to control the computing module to execute task processing according to the resource allocation strategy; An output synchronization control module connected to the computing module and configured to: receive the first priority task processing result, the second priority task processing result, and the third priority task processing result output by the computing module, synthesize the radar distance analysis data and the vehicle outline data through a hardware superposition layer, perform frame rate synchronization operation on the third priority task processing result, and output a synchronization processing result to a display device; A fault tolerance module connected to the genetic algorithm optimization scheduling module and the computing module and configured to: monitor the iteration cycle length of the genetic algorithm instance, send a hardware reset instruction to the computing module when the iteration cycle length exceeds a preset timeout threshold, and inject a degraded scheduling strategy to replace the current resource allocation strategy.

[0015] The application has the following advantages: The application has the beneficial effects that: the multi-channel video acquisition module eliminates the time sequence difference of multi-source video streams through signal standardization and timestamp alignment operation, and provides a time sequence unified data source for task classification; the classification calculation task decoupling module injects priority labels into tasks based on resolution and signal source characteristics, establishes a task mapping system with radar analysis as the first priority, vehicle outline generation as the second priority, and video fusion as the third priority; the calculation module combines the genetic algorithm to optimize the dynamic resource allocation instruction of the scheduling module, executes differential processing according to the label priority, and guarantees the calculation resources of high-priority tasks; the genetic algorithm optimization scheduling module generates a resource allocation strategy through historical scene simulation and fitness function dynamic evaluation to reduce the maximum delay of tasks; the output synchronization control module uses a hardware overlay layer and a vertical synchronization trigger mechanism to accelerate the output of key information, and feeds back delay data to optimize the scheduling weight; the fault tolerance module switches to a degradation strategy after iteration timeout to maintain the basic functions of the system, and the whole system cooperates through task classification processing, dynamic resource scheduling and hardware acceleration to solve the real-time degradation problem caused by multi-channel high-resolution video stream processing and reduce the image output delay in the auxiliary driving scene. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on the drawings.

[0017] Figure 1 The system architecture diagram of the auxiliary driving system based on video image processing provided by the embodiment of the present application.

[0018] Figure 2 The video display position and scale diagram in the display provided by the embodiment of the present application.

[0019] Figure 3 The radar distance superposition position diagram provided by the embodiment of the present application.

[0020] Figure 4 The system power supply and control architecture provided by the embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the technical solutions of the present application clearer, the following will combine the specific embodiments of the present application and the corresponding drawings to clearly and completely describe the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. The following will combine the drawings to specifically describe the present application provided by the embodiments of the present application. In order to better understand the purpose of the present application, the following will further describe the present application in detail.

[0022] In a first aspect, the application provides an auxiliary driving device based on video image processing, comprising: a multi-channel video acquisition module, which synchronously receives multiple external video signals, performs standardized conversion on the video signals, and outputs a frame buffer queue with timestamp alignment; a hierarchical computing task decoupling module, which receives the frame buffer queue of the multi-channel video acquisition module, identifies the task type of the video data in the frame buffer queue, marks a priority label according to a preset real-time rule, the priority label includes: marking the radar data analysis and distance superposition task as a first priority label; marking the vehicle outline generation task as a second priority label; marking the video fusion task as a third priority label; and outputting a data stream with the priority label; a computing module, configured to receive the data stream with the priority label output by the hierarchical computing task decoupling module and the resource allocation instruction sent by the genetic algorithm optimization scheduling module; processing the data stream corresponding to the first priority label to generate a first priority task processing result; processing the data stream corresponding to the second priority label to generate a second priority task processing result; processing the data stream corresponding to the third priority label to generate a third priority task processing result; a genetic algorithm optimization scheduling module, which receives the data stream with the priority label of the hierarchical computing task decoupling module, and performs: running multiple groups of genetic algorithm instances in parallel to generate resource allocation strategies, the genetic algorithm instance includes chromosome coding and fitness function, the chromosome coding represents the processing resource allocation ratio and task sequence; loading historical scene data to simulate video processing task flow, and evaluating the optimization effect of the resource allocation strategy on task delay; selecting the resource allocation strategy with the highest fitness, and sending the resource allocation instruction generated based on the resource allocation strategy to the computing module, the resource allocation instruction is used to control the computing module to execute task processing according to the resource allocation strategy; an output synchronization control module, which receives the first priority task processing result and the second priority task processing result output by the computing module, and receives the third priority task processing result output by the computing module; the output synchronization control module synthesizes the radar distance analysis data in the first priority task processing result and the vehicle outline data in the second priority task processing result through a hardware superposition layer; performs frame rate synchronization operation on the third priority task processing result, and outputs a synchronization processing result to a display device.

[0023] The system power supply and control architecture adopted in the embodiment of the application is described in Figure 4The power module board provides redundant power supply for the whole system, the power control board receives the instruction of the genetic algorithm optimization scheduling module to switch the power supply path (such as closing the front view unit power supply in the reverse mode), and the video processing board carries the functions of the hierarchical computing task decoupling module, the computing unit and the genetic algorithm optimization scheduling module.

[0024] The multi-channel video acquisition module synchronously receives multiple external video signals, including AHD high-definition video signals and CVBS infrared video signals. The analog high-definition signal is subjected to signal conversion operation to output digital video stream; the CVBS infrared signal is subjected to USB video protocol unpacking operation to output YUV422 format data. A double buffering mechanism is adopted to cache input frames, and the next frame is stored in an independent storage area while the current frame is processed. An external clock source is used to perform timestamp alignment operation on the video stream data in the independent storage area, and the timestamps of the digital video stream and the YUV422 format data corresponding to the video stream are formed into a timestamp-aligned frame buffer queue.

[0025] The hierarchical computing task decoupling module receives the video data in the frame buffer queue, parses the metadata of the video data to obtain resolution parameters and signal source types. According to the analysis results of the resolution parameters and the signal source types, the video data is classified into radar data stream, rear view video stream and front view fusion video stream. The first priority label is injected into the classified radar data stream, the second priority label is injected into the classified rear view video stream, and the third priority label is injected into the classified front view fusion video stream, forming a data stream with priority labels and outputting to the computing module.

[0026] The computing module receives the data stream with priority labels output by the hierarchical computing task decoupling module and the resource allocation instruction. The data stream corresponding to the first priority label is processed to generate the first priority task processing result, the data stream corresponding to the second priority label is processed to generate the second priority task processing result, and the data stream corresponding to the third priority label is processed to generate the third priority task processing result. The execution process of the computing module is controlled by the resource allocation instruction, and the processing resources are allocated according to the resource allocation strategy to execute the task processing operation.

[0027] The genetic algorithm optimization scheduling module receives the priority-labeled data stream from the hierarchical computing task decoupling module. Multiple sets of genetic algorithm instances are run in parallel to generate resource allocation strategies, including chromosome encoding and fitness function, which represents the processing resource allocation ratio and task sequence. A preset historical scenario dataset is loaded to generate multiple video concurrent peak loads based on the historical scenario dataset, and the priority-labeled data stream is injected into the simulation environment. The resource allocation strategy is applied to process the data stream. The completion time delay of the first priority task, the second priority task, and the third priority task under the resource allocation strategy is recorded, and the recorded completion time delay is input into the fitness function to calculate the evaluation value. The fitness function variables include the maximum delay of the task and the resource utilization rate, and the function weight coefficient is dynamically adjusted according to the processor load rate and the memory occupancy rate. The resource allocation strategy with the highest fitness is selected, and a resource allocation ratio instruction is sent to the computing module.

[0028] The output synchronization control module receives the first priority task processing result, the second priority task processing result, and the third priority task processing result output by the computing module. The radar range resolution data in the first priority task processing result and the vehicle outline line data in the second priority task processing result are synthesized through a hardware overlay layer. Frame rate synchronization operation is performed on the third priority task processing result, and direct memory access transmission of the third priority task processing result in the display memory is triggered when the vertical synchronization signal is at the starting point of the vertical blanking period. The actual image output delay of the high-definition multimedia interface is monitored, and the actual image output delay is fed back to the fitness function weight adjustment parameter of the genetic algorithm optimization scheduling module.

[0029] The data flow between the modules forms a closed loop: the video signal is standardized into a frame buffer queue by the acquisition module; the video data is classified and labeled with priority by the decoupling module; the data stream is processed by the computing module according to the priority label to generate task results; the genetic algorithm module dynamically optimizes the resource allocation strategy; and the output module synthesizes the results and feeds back the delay parameters. The technical solution solves the real-time degradation problem caused by the computing load of multiple high-resolution video streams through hierarchical processing of priority tasks and dynamic scheduling of genetic algorithms.

[0030] Specifically, the auxiliary driving device based on video image processing of the application is configured as: receiving an analog high-definition signal and performing signal conversion processing to output a digital video stream; receiving a composite video blanking synchronization infrared signal and implementing data unpacking to output YUV color coding format data through a universal serial bus protocol; The multi-channel video acquisition module generates a frame buffer queue, including: receiving a digital video stream and YUV422 format data, using a double buffering mechanism to cache input frames, and storing the next frame in an independent storage area while the current frame is being processed; The timestamp alignment operation is performed on the video stream data in the independent storage area, and the digital video stream is aligned with the video stream timestamp corresponding to the YUV422 format data by using an external clock source to form a frame buffer queue.

[0031] The multi-channel video acquisition module receives an externally input AHD high-definition video signal, converts the analog high-definition signal into a BT.656 standard digital stream through a dedicated video decoding chip.

[0032] When the frame buffer queue is generated, the module receives a digital video stream and YUV422 format data as input sources.

[0033] The timestamp alignment operation is performed on the video stream data in the independent storage area.

[0034] The frame buffer queue output includes two core features: the timestamp is aligned to the microsecond error range, and the video format is unified to the YUV420 plane format.

[0035] Specifically, the auxiliary driving device based on video image processing is configured as: The task type recognition manner of the hierarchical computing task decoupling module includes: Receiving video data in the frame buffer queue, parsing the metadata of the video data to obtain resolution parameters and signal source types; According to the resolution parameters and the signal source type analysis result, the video data is classified into radar data stream, rearview video stream and frontview fusion video stream; Injecting a first priority label into the classified radar data stream, injecting a second priority label into the classified rearview video stream, and injecting a third priority label into the classified frontview fusion video stream to form a data stream with priority labels.

[0036] The hierarchical computing task decoupling module receives the frame buffer queue output by the multi-channel video acquisition module. The metadata of the video data in the frame buffer queue is parsed, and the resolution parameter and the signal source type are extracted. The resolution parameter includes the horizontal pixel number and the vertical pixel number; the signal source type is identified as an AHD camera, a CVBS infrared sensor or a radar interface. The metadata parsing is realized based on the video stream header information decoding, and the physical characteristics of different data sources are distinguished.

[0037] According to the resolution parameter and the signal source type, the video data is classified into radar data stream, rear-view video stream and front-view fusion video stream. The classification rule is: when the signal source type is a radar interface and the resolution is lower than the threshold, it is classified as a radar data stream; when the signal source type is an AHD camera and the resolution meets the high-definition standard, it is classified as a rear-view video stream; when the signal source type is a CVBS infrared sensor and the resolution matches the preset range, it is classified as a front-view fusion video stream. The classification process realizes data label binding through OpenCL kernel, and generates a classification intermediate result.

[0038] The first priority label is injected into the classified radar data stream, and the label code is binary value 0x01. The second priority label is injected into the classified rear-view video stream, and the label code is 0x02. The third priority label is injected into the classified front-view fusion video stream, and the label code is 0x04. The label injection operation is realized by rewriting the video stream metadata field, and the priority identifier is embedded in the data stream header.

[0039] The priority-labeled data stream includes three types of data entities: the radar data stream carrying the 0x01 label carries distance analysis instructions; the rear-view video stream carrying the 0x02 label includes vehicle contour coordinate information; and the front-view fusion video stream carrying the 0x04 label includes infrared and low-light fusion frame data. The priority-labeled data stream is output to the computing module and the genetic algorithm optimization scheduling module for subsequent differentiated processing.

[0040] Specifically, the auxiliary driving device based on video image processing of the application is configured as: The process of the genetic algorithm optimization scheduling module executing simulated video processing task flow includes: Load the preset historical scene data set, generate multi-path video concurrent peak load based on the historical scene data set; Inject the priority-labeled data stream into the preset virtualization environment, and apply a resource allocation strategy to process the priority-labeled data stream; Record the completion delay of the first priority task, the second priority task and the third priority task under the resource allocation strategy; The recorded completion time delay is input into an adaptive function to calculate an evaluation value, and the adaptive function variables include task maximum delay and resource utilization, and the function weight coefficient is dynamically adjusted according to the processor load rate and the memory occupancy rate.

[0041] The genetic algorithm optimization scheduling module loads a preset historical scene data set, and the data set includes multi-path video concurrent peak load parameters in a typical driving scene. The historical scene data covers video stream features in low-light reversing, high-speed cruising and complex intersection scenes. The reproduction process simulates nine-path video input concurrent pressure and restores the calculation resource contention state in the actual road condition.

[0042] The priority-labeled data stream output by the decoupling module of the hierarchical calculation task is injected into the simulation environment. The simulation environment is pre-built on a virtualization platform and includes a processor core architecture consistent with the physical calculation module. The priority-labeled data stream is processed by applying a resource allocation strategy, which includes a processor core ratio and a task scheduling sequence defined by chromosome coding. The processing process completely reproduces the priority task execution pipeline.

[0043] The completion time delay of the first priority task, the second priority task and the third priority task under the resource allocation strategy is recorded. The recording process monitors the task start timestamp and the result output timestamp, and calculates the time difference value as the single task time delay. The maximum time delay, the average time delay and the time delay variance of each priority task in the simulation environment are counted to form a time delay index set.

[0044] The recorded completion time delay index set is input into an adaptive function to calculate an evaluation value. The adaptive function adopts a multi-dimensional weighted calculation model, and the function variables include task maximum delay and resource utilization. The function weight coefficient is dynamically adjusted according to the real-time system state: when the system detects a high load state, the task maximum delay weight is increased; when the system is in a low load state, the resource utilization weight is increased. The evaluation value quantifies the comprehensive performance of the resource allocation strategy.

[0045] Specifically, the auxiliary driving device based on video image processing of the present application, the genetic algorithm optimization scheduling module is further configured to: The genetic algorithm optimization scheduling module performs a strategy switching operation in response to an external control signal, including: in response to the received vehicle gear signal or bus instruction; When the vehicle gear signal or bus instruction represents the reversing mode, send an instruction to the calculation module to pause processing the data stream corresponding to the third priority task, and reallocate the released calculation resources to the first priority task and the second priority task; When the vehicle gear signal or bus instruction represents the forced radar mode, send an instruction to the calculation module to make the first priority task exclusively occupy the processing resource proportion defined by the resource allocation strategy; The resource allocation strategy is a resource allocation strategy whose fitness evaluation value reaches a preset threshold.

[0046] The genetic algorithm optimization scheduling module receives a vehicle gear signal or a vehicle-mounted bus instruction through a CAN bus interface. The signal analysis module extracts instruction features: the gear signal includes forward gear, reverse gear and parking gear state codes; the bus instruction includes a forced radar mode trigger identifier.

[0047] When the signal analysis result represents a reverse mode, the genetic algorithm optimization scheduling module sends a task suspension instruction to the calculation module. The instruction format is a 32-bit control word, the upper 16 bits specify the third priority task identifier, and the lower 16 bits define the suspension operation code. The calculation module stops processing the data stream corresponding to the third priority task after responding to the instruction, releases the occupied processing core and memory bandwidth. The released computing resources are reallocated to the first priority task and the second priority task processing flow according to the resource allocation strategy.

[0048] When the signal analysis result represents a forced radar mode, the module calls the pre-stored resource allocation strategy with the highest fitness. Send an exclusive control instruction to the calculation module, which includes the first priority task identifier and the processing resource proportion defined by the resource allocation strategy. The calculation module configures a dedicated processing core according to the instruction, prohibits other tasks from occupying the specified proportion of computing resources, and makes the first priority task exclusive of the execution environment.

[0049] The resource allocation strategy is derived from the historical evaluation results of the genetic algorithm optimization scheduling module. The resource allocation strategy with the highest fitness is stored in a non-volatile memory, including processor core allocation proportion, memory channel occupancy rate and task execution sequence code. The strategy switching process maintains the integrity of the chromosome code, and realizes the scheduling logic that meets the real-time constraints.

[0050] Specifically, the auxiliary driving device based on video image processing of the application is configured as: The hardware overlay layer of the output synchronization control module runs in the following modes: Receiving radar distance analysis data in the first priority task processing result, calling a preset video processing engine to write the radar distance analysis data in pixel format to the hardware overlay layer of the display controller; Receiving the third priority task processing result and storing it in the video memory; When the vertical synchronization signal of the display device is at the starting point of the vertical blanking period, triggering the physical transmission channel of the direct memory access controller to transmit the third priority task processing result in the video memory; The actual image output delay of the high-definition multimedia interface is monitored, and the actual image output delay is fed back to the fitness function weight adjustment parameter of the genetic algorithm optimization scheduling module.

[0051] The output synchronization control module receives the first priority task processing result output by the calculation module. Radar range analysis data including obstacle coordinates and distance parameters are extracted from the first priority task processing result. The radar range analysis data is converted into an ARGB8888 format pixel block by calling the hardware overlay layer interface of the special video processing engine. The converted data is written into the display overlay layer video memory address through the DMA channel, and the overlay layer is independent of the main video rendering pipeline.

[0052] The third priority task processing result output by the calculation module is received, and the third priority task processing result is video fusion frame data. The video fusion frame data is stored in the background buffer area of the GPU video memory double buffer area. The video memory area pre-allocates storage space according to resolution and color depth to avoid delay caused by dynamic memory allocation.

[0053] The display controller monitors the vertical synchronization signal state. When the rising edge of the vertical synchronization signal arrives, the direct memory access controller is triggered to start the transmission process. The direct memory access controller reads the video fusion frame data in the video memory background buffer area and transmits it to the frame buffer area of the HDMI transmitter through the hardware channel. The transmission process skips the CPU intervention, reducing the data movement delay.

[0054] The actual image output delay of the high-definition multimedia interface is monitored in real time. The delay measurement records the time interval from the video frame entering the video memory to the completion of the output from the HDMI physical port. The actual image output delay value is written into the feedback register of the shared memory. The genetic algorithm optimization scheduling module periodically reads the feedback register value to dynamically adjust the fitness function weight adjustment parameter. When the weight adjustment parameter increases, the influence of the maximum delay of the task on the resource allocation evaluation increases.

[0055] Specifically, the auxiliary driving device based on video image processing of the application further comprises: When the system detects a sensor missing event: The hierarchical calculation task decoupling module receives the sensor missing signal and marks the task corresponding to the missing sensor as invalid state; The genetic algorithm optimization scheduling module receives the invalid state mark and updates the weight parameter of the task corresponding to the invalid state mark in the fitness function; The resource allocation strategy is regenerated based on the updated fitness function, and a resource recovery instruction is sent to the computing module, the instruction including a processing resource identifier of the task marked with the invalid state, the computing module releasing memory allocation and cache space of the specified processing core according to the instruction, and the recovered computing resource being allocated to the valid task processing flow, the resource allocation ratio complying with the newly generated resource allocation strategy definition.

[0056] The system monitors the sensor connection state through a hardware probe. When the sensor power supply is interrupted or the communication timeout lasts for more than a preset period, it is determined that a sensor loss event has occurred. The sensor loss signal includes a sensor type identifier and a fault code, and the signal format complies with the CAN bus error frame standard.

[0057] The hierarchical computing task decoupling module receives the sensor loss signal. According to the sensor type identifier, a preset mapping table is queried to determine the task type associated with the missing sensor. The task corresponding to the missing sensor is marked as an invalid state, and the invalid state mark is written in the form of a binary flag bit in the data stream metadata header. The marked data stream retains the original data structure but carries an invalid identification.

[0058] The genetic algorithm optimization scheduling module receives the data stream marked with the invalid state. The type identification of the task corresponding to the invalid state mark is parsed, and the weight parameter configuration of the fitness function is updated. In the fitness function calculation model, the weight parameters of the task corresponding to the invalid state mark are shielded, so that the resource allocation strategy ignores the related task indicators. Based on the updated fitness function, a new genetic algorithm instance is run to generate a new resource allocation strategy.

[0059] A resource recovery instruction is sent to the computing module, the instruction including a processing resource identifier of the task marked with the invalid state. The computing module releases the memory allocation and cache space of the specified processing core according to the instruction. The recovered computing resource is allocated to the current valid task processing flow, and the resource allocation ratio complies with the newly generated resource allocation strategy definition.

[0060] Specifically, the auxiliary driving device based on video image processing of the application is configured, and the genetic algorithm optimization scheduling module is further configured to: The genetic algorithm optimization scheduling module acquires external sensor data and determines a preset distance range boundary value; In the video fusion task processing process, when the input data is within the preset distance range boundary value, a high-precision registration algorithm is executed; when the input data exceeds the preset distance range boundary value, a fast alignment algorithm is executed; The ambient light intensity is detected by a photosensitive sensor, and when the light intensity is lower than 50 lux, resolution down-sampling processing is performed on the collected video data to generate down-sampled frame data, the down-sampled frame data is input into the third priority task processing flow, and the third priority task processing result is generated.

[0061] The genetic algorithm optimization scheduling module obtains external sensor data through the vehicle bus, and the sensor data includes distance measurement values of a laser range finder or a millimeter wave radar. A preset distance range boundary value is determined based on a statistical analysis result of the distance measurement values, and the boundary value divides a distance threshold of a near-field region and a far-field region. The distance range boundary value is stored in a configuration register for real-time calling.

[0062] In the video fusion task processing process, the module detects the spatial coordinate information of the input data. When the input data coordinates are located in the near-field region defined by the preset distance range boundary value, a high-precision registration algorithm is executed. The high-precision registration algorithm uses feature point matching and perspective transformation matrix calculation to realize sub-pixel level alignment of multi-source video streams. When the input data coordinates exceed the far-field region defined by the boundary value, a fast alignment algorithm is executed. The fast alignment algorithm is based on phase correlation and block matching technology, and reduces the computational complexity to meet the real-time constraint.

[0063] For video data collected in an environment with light intensity detected by a photosensitive sensor lower than 50 lux, a resolution downsampling operation is performed. The downsampling operation uses a bilinear interpolation algorithm to convert the original resolution video frame into downsampled frame data. The downsampled frame data retains key feature information but reduces the number of pixels, and the output data format matches the input specification of the video fusion task processing flow. The downsampled frame data is directly input to the processing queue of the video fusion task processing flow through the memory channel.

[0064] Specifically, the auxiliary driving device based on video image processing of the application further comprises a fault tolerance module connected to the genetic algorithm optimization scheduling module, and the fault tolerance module is configured to: monitor the iteration period of the genetic algorithm instance of the genetic algorithm optimization scheduling module, and record the iteration period length; when the iteration period length exceeds a preset timeout threshold, send a hardware reset instruction to the computing module; load a pre-stored degraded scheduling strategy to replace the current resource allocation strategy, and inject the degraded scheduling strategy into the computing module for execution.

[0065] The fault tolerance module monitors the iteration period of the genetic algorithm instance of the genetic algorithm optimization scheduling module through a hardware timer. The complete iteration period length from the generation of the initial population to the completion of the fitness evaluation of each genetic algorithm instance is recorded. The iteration period length data is stored in a circular buffer, and the history of the last ten iterations is retained.

[0066] When the iteration cycle length exceeds the preset timeout threshold, the fault-tolerant module sends a hardware reset instruction to the computing module through the system bus. The hardware reset instruction includes three levels of reset signals: a first level reset to clear the computing module pipeline task; a second level reset to recycle memory allocation resources; and a third level reset to reset the processor core state register. The reset operation forcibly terminates the execution of non-real-time tasks.

[0067] A pre-stored degradation scheduling strategy in the non-volatile memory is loaded. The degradation scheduling strategy is a fixed resource allocation scheme, and the priority order is a first priority task, a second priority task, and a third priority task. The strategy register of the genetic algorithm optimization scheduling module is rewritten, and the current resource allocation strategy is replaced with the degradation scheduling strategy. The degradation scheduling strategy is injected into the computing module execution environment through a memory mapping interface to overwrite the original strategy configuration parameters.

[0068] In a second aspect, referring to Figure 1 The application provides an auxiliary driving system based on video image processing, which is applied to an auxiliary driving device based on video image processing and includes: A multi-channel video acquisition module configured to synchronously receive multiple external video signals, perform standardized conversion on the video signals, and output a frame buffer queue with timestamp alignment; A hierarchical computing task decoupling module connected to the multi-channel video acquisition module and configured to: receive the frame buffer queue, identify the task type of the video data, and mark a priority label according to a preset real-time rule, including marking a radar data analysis and distance superposition task as a first priority label, marking a vehicle outline generation task as a second priority label, and marking a video fusion task as a third priority label, and output a data stream with the priority label; A computing module connected to the hierarchical computing task decoupling module and configured to: receive the data stream with the priority label and a resource allocation instruction, process the data stream corresponding to the first priority label to generate a first priority task processing result, process the data stream corresponding to the second priority label to generate a second priority task processing result, and process the data stream corresponding to the third priority label to generate a third priority task processing result; A genetic algorithm optimization scheduling module connected to the hierarchical computing task decoupling module and the computing module and configured to: receive the data stream with the priority label, run multiple groups of genetic algorithm instances in parallel to generate a resource allocation strategy, load historical scene data to simulate a video processing task flow, and send a resource allocation instruction generated based on the resource allocation strategy to the computing module, where the resource allocation instruction is used to control the computing module to execute task processing according to the resource allocation strategy; The output synchronization control module is connected to the calculation module and is configured to receive the first-priority task processing result, the second-priority task processing result and the third-priority task processing result output by the calculation module, synthesize radar range resolution data and vehicle outline line data through a hardware overlay layer, perform frame rate synchronization operation on the third-priority task processing result, and output the synchronization processing result to a display device. Figure 2 As shown in the figure, the front view fusion image area of the display device is allocated with 1190*720 pixels, the left side view and the right side view area each occupies 360*357, and the rear view area is 720*576. The radar range resolution data is written into the front view area specified coordinates in ARGB format through a special video processing engine, and the vehicle outline line data is synthesized to the bottom auxiliary line layer of the rear view area. The multi-picture segmentation ratio and the vertical synchronization signal triggering mechanism cooperate to realize real-time superposition display of high-priority task processing results. Figure 3 It is further clarified that the rear radar range information is displayed at the bottom central position of the rear view image, the coordinate mapping hardware overlay layer writes the address, and the sub-pixel level rendering of the obstacle distance information is realized.

[0069] The fault tolerance module is connected to the genetic algorithm optimization scheduling module and the calculation module, and is configured to monitor the iteration period length of the genetic algorithm instance, and when the iteration period length exceeds a preset timeout threshold, send a hardware reset instruction to the calculation module and inject a degraded scheduling strategy to replace the current resource allocation strategy.

[0070] The multi-channel video acquisition module synchronously receives multiple external video signals, including AHD high-definition signals and CVBS infrared signals. The analog high-definition signal is subjected to signal conversion operation to output digital video stream; the CVBS infrared signal is subjected to universal serial bus protocol unpacking operation to output YUV422 format data. A double buffering mechanism is used to cache input frames, and the next frame is stored in an independent storage area while the current frame is being processed. The GPS second pulse signal is used as an external clock source to perform timestamp alignment operation on the video stream data in the independent storage area, to calibrate the timestamps of the nine video signals to microsecond level accuracy, and form a standardized frame buffer queue output.

[0071] The hierarchical calculation task decoupling module receives video data in the frame buffer queue. The video metadata is parsed to obtain resolution parameters and signal source types. The resolution parameters include horizontal pixel number and vertical pixel number. According to the parsing result, the video data is classified: the signal source is radar interface and the resolution is lower than the threshold, which is classified as radar data stream; the signal source is AHD camera, which is classified as rear view video stream; the signal source is CVBS infrared sensor, which is classified as front view fusion video stream. The first-priority label 0x01 is injected into the radar data stream, the second-priority label 0x02 is injected into the rear view video stream, and the third-priority label 0x04 is injected into the front view fusion video stream, forming a data stream with priority label output to the calculation module.

[0072] The computing module receives a data stream with a priority label and a resource allocation instruction. The data stream corresponding to the first priority label 0x01 is processed to generate radar range resolution data. The data stream corresponding to the second priority label 0x02 is processed to generate vehicle outline line data. The data stream corresponding to the third priority label 0x04 is processed to generate video fusion frame data. The processing process allocates processor cores and memory bandwidth according to the resource allocation instruction, and performs priority task processing operations.

[0073] The genetic algorithm optimization scheduling module receives a data stream with a priority label. Multiple genetic algorithm instances are run in parallel to generate resource allocation strategies. The chromosome encoding of the genetic algorithm instance includes a CPU / NPU resource allocation ratio bitmap. Historical scene data sets are loaded to generate multiple video concurrent peak loads based on the historical scene data sets. The data stream with the priority label is injected into the simulation environment to test the performance of the strategy. The resource allocation instruction is sent to the computing module to control the task processing process. The instruction includes processor core allocation ratio and task execution sequence encoding.

[0074] The output synchronization control module receives three types of task processing results output by the computing module. The radar range resolution data is converted to ARGB8888 format and written to the display overlay layer through the hardware overlay layer. The video fusion frame data is stored in the video memory background buffer. When the vertical synchronization signal is at the starting point of the vertical blanking period, the direct memory access transmission is triggered, and the video memory data is transmitted to the HDMI transmitter. The actual output delay of the high-definition multimedia interface is monitored and fed back to the genetic algorithm module.

[0075] The fault tolerance module monitors the iteration period length of the genetic algorithm instance. When the iteration period length exceeds the preset timeout threshold, a three-level reset instruction is sent to the computing module to clear the processing state. The degraded scheduling strategy in the non-volatile memory is loaded, the current resource allocation strategy is replaced through the strategy register, and the fixed priority resource allocation scheme is injected into the computing module execution environment.

[0076] The technical scheme of the application solves the high-resolution video stream processing delay problem through a multi-level cooperative mechanism. The multi-channel video acquisition module synchronously receives external video signals, performs standardized conversion, and outputs timestamp-aligned frame buffer queues to eliminate the timing differences of multi-source heterogeneous signals. The hierarchical computing task decoupling module analyzes video metadata and injects priority labels according to real-time requirements: radar data analysis and range overlay tasks are marked with the first priority label, vehicle outline generation tasks are marked with the second priority label, and video fusion tasks are marked with the third priority label. The priority label establishes a mapping relationship between tasks and computing resources.

[0077] The computing module receives the data stream with priority labels, combines the resource allocation instruction issued by the genetic algorithm optimization scheduling module, and executes differential processing according to the label priority. The genetic algorithm optimization scheduling module runs multiple genetic algorithm instances in parallel to generate resource allocation strategies, simulates video processing task flow through historical scene data, and evaluates the optimization effect of the strategy on task delay. The fitness function dynamically adjusts the weight coefficients of the maximum task delay and resource utilization, and selects the optimal strategy to control the resource allocation of the computing module.

[0078] The output synchronization control module synthesizes high-priority task processing results through a hardware overlay layer, and performs frame rate synchronization operations on low-priority video fusion tasks. The vertical synchronization signal triggers direct memory access to transfer video memory data, and hardware acceleration reduces output delay. The key improvement is that when the system detects the reverse mode, the genetic algorithm module suspends the third-priority task processing, and releases the resources to reallocate to high-priority tasks; when the sensor is missing, the resource strategy is dynamically updated to recover invalid task resources; the video fusion task switches the registration algorithm according to the distance from the boundary value, and uses downsampling operation in low-illumination environment to reduce the computational load. Task grading and dynamic resource scheduling form a closed loop, ensuring the execution of critical tasks, and controlling the image output delay within the response threshold.

[0079] The specific implementation steps of the present application are as follows: The multi-channel video acquisition module receives eight AHD high-definition video signals and one CVBS infrared video signal through a dedicated interface. TW9990 decoding chip is used to convert analog high-definition signals into digital video streams, and USB video protocol is used to unpack CVBS infrared signals to output YUV422 format data. Double buffering mechanism is used to manage input frames: when the current frame enters the processing flow, the next frame is stored in the independent storage area divided by DDR3 memory. GPS second pulse triggers hardware interrupt, and timestamp alignment operation is performed on nine video streams, with calibration accuracy reaching microseconds, forming a standardized frame buffer queue output. This process eliminates the timing differences of multiple source signals, providing time-consistent data sources for task grading.

[0080] The hierarchical computing task decoupling module parses the metadata of video data in the frame buffer queue, extracts the resolution parameters and signal source types. According to the preset classification rules: the resolution is lower than 1280x720 and the signal source is radar interface, it is classified as radar data stream; the resolution is greater than or equal to 1920x1080, and the analog high-definition signal is classified as rearview video stream; the rest of the CVBS infrared signal is classified as forward fusion video stream. The first priority label 0x01 is injected into the radar data stream, the second priority label 0x02 is injected into the rearview video stream, and the third priority label 0x04 is injected into the forward fusion video stream, generating data streams with priority labels. Label encoding is realized by rewriting the metadata field of the data stream header, establishing accurate mapping between tasks and real-time requirements.

[0081] The computing module receives the data stream with priority tags and the resource allocation instruction issued by the genetic algorithm optimization scheduling module. The first priority tag 0x01 corresponding data stream is processed to generate radar range resolution data; the second priority tag 0x02 corresponding data stream is processed to generate vehicle outline line coordinates; and the third priority tag 0x04 corresponding data stream is processed to output video fusion frames. The resource allocation instruction controls the computing module to allocate processing cores according to the strategy, for example, in the reversing mode, the resources are concentrated to the first two types of tasks. The task processing result retains the original data structure, and the priority identifier is carried when output.

[0082] The genetic algorithm optimization scheduling module loads the historical scene data set to simulate the night reversing working condition, and reproduces the nine-way video 25fps concurrent load. The data stream with priority tags is injected into the virtualization test environment, and the resource allocation strategy defined by the chromosome coding is applied to process the data. The delay of each priority task is recorded: the average delay of radar resolution task, the peak delay of vehicle outline line generation, and the delay variance of video fusion task. The fitness function takes the maximum delay and resource utilization rate as variables, and the weight coefficient is dynamically adjusted according to the system load. The strategy with the highest fitness is selected and sent to the computing module through the CAN bus, and the response time of strategy switching is less than 5ms.

[0083] The output synchronization control module calls the IVE engine of Hi3559A, writes the radar range resolution data into the HDMI overlay layer video memory address in ARGB8888 format. The video fusion frame is stored in the GPU video memory double buffer area, and the rising edge of the vertical synchronization signal triggers DMA transmission to the HDMI transmitter. The delay from frame data into video memory to HDMI physical output is monitored in real time, and the feedback value is written into the shared register of the genetic algorithm module, driving the dynamic optimization of the fitness function weight.

[0084] The fault tolerance module monitors the genetic algorithm iteration period, and sends three-level reset instructions to the computing module after timeout: the first level clears the pipeline task, the second level recycles memory resources, and the third level resets the core register. Load the pre-stored degradation scheduling strategy in ROM to cover the strategy register of the genetic algorithm module, and fix the resource allocation proportion as 45% for the first priority task, 35% for the second priority task, and 20% for the third priority task.

[0085] In the night reversing scene verification, the system responds to the external gear signal switching to the reversing mode. The genetic algorithm module suspends the third priority task processing, and distributes the released NPU resource to the radar analysis and vehicle outline line generation task. When the side radar fails, the decoupling module marks the associated task invalid, the scheduling module updates the function to shield the invalid task weight, and the calculation module recycles the resource and redistributes it to the valid task. The video fusion task adopts SIFT feature matching high-precision registration within 10 meters, and switches to fast phase alignment beyond the range, and low-illuminance video is down-sampled to 640x480 for processing. After hardware acceleration output, the obstacle warning delay is compressed to within 50ms, solving the problem of video fusion blocking key warning Embodiment one of the present application; In the night reversing scene, the multi-channel video acquisition module receives eight AHD high-definition rearview video signals and one CVBS infrared video signal. A special decoding chip converts the analog high-definition signal into a digital video stream, and a universal serial bus protocol unpacks the CVBS signal to output YUV422 data. Double buffering mechanism is adopted to cache input frames, and GPS second pulse triggers hardware interrupt to align nine video timestamps to microsecond level accuracy, forming a standardized frame buffer queue. The hierarchical calculation task decoupling module analyzes the video metadata: radar interface data with resolution lower than 720p is classified as a radar data stream and labeled with a first priority tag; analog high-definition signals of 1920x1080 are classified as rearview video streams and labeled with a second priority tag; the remaining infrared signals are classified as front view fusion video streams and labeled with a third priority tag.

[0086] The genetic algorithm optimization scheduling module detects the reversing gear signal, sends an instruction to the calculation module to suspend processing of the third priority tag data stream, and releases the NPU resource. The resource is redistributed to the radar analysis task of the first priority tag and the vehicle outline line generation task of the second priority tag. The calculation module processes the radar data stream to generate obstacle distance information according to the resource allocation instruction, and processes the rearview video stream to generate real-time auxiliary lines. The output synchronization control module directly writes the radar distance data into the display layer in ARGB format through the Hi3559A hardware overlay layer, and simultaneously triggers DMA transmission of the video fusion frame through the vertical synchronization signal. The system completes the superposition of obstacle warning information within 50ms, solving the low-speed collision risk caused by the video fusion task blocking.

[0087] Embodiment two of the present application; Under the urban peak congestion scenario, the genetic algorithm optimization scheduling module loads the historical data set to simulate nine video 25fps concurrent load. The priority tagged data stream is injected into the virtual environment to test the resource allocation strategy: the chromosome coding defines the resource allocation ratio of CPU 60%, NPU 30% and GPU 10%. The maximum delay of the first priority task, the average delay of the second priority task and the delay variance of the third priority task are recorded. The fitness function dynamically adjusts the weight according to the real-time load: the maximum delay weight of the task is increased to 0.7 under high load, and the resource utilization weight is increased to 0.6 under low load. The optimal strategy is selected to control the calculation module to execute task processing.

[0088] When the millimeter wave radar detects that the distance of the front vehicle is less than 10 meters, the video fusion task is switched to the high-precision SIFT registration algorithm; when the distance is greater than 30 meters, the fast phase alignment is enabled. In a low-illumination environment, the CVBS infrared video is executed to reduce the resolution to 640x480 to generate a down-sampling frame. When the genetic algorithm iteration period is detected to be overdue, the fault tolerance module immediately sends a three-level reset instruction to the calculation module: clears the pipeline task, recycles memory resources and resets the core register. Load the degraded scheduling strategy in the ROM, and allocate resources in a fixed proportion (radar analysis 45%, auxiliary line generation 35% and video fusion 20%). Through dynamic resource scheduling and algorithm optimization, the image output delay is stably controlled within the 100ms threshold under multi-task concurrency, eliminating the complex road condition warning lag problem.

[0089] The technical features of the present application are explained as follows: The multi-channel video acquisition module realizes the physical layer integration of multi-source video signals, converts different format video streams into standard digital format through a dedicated decoding chip, and eliminates signal source differences; the double buffering mechanism cooperates with the external clock source to realize the microsecond level timestamp alignment of multiple video streams, solving the multi-sensor data timing dislocation problem. The hierarchical computing task decoupling module classifies data streams according to resolution parameters and signal source types, binds radar data streams, rear-view video streams and front-view fusion video streams with differentiated priority tags based on metadata analysis results, and constructs a task real-time hierarchical system.

[0090] The calculation module is a task execution carrier, which processes corresponding data streams according to the priority tag to generate task results, and dynamically controls the allocation proportion of the processing core through the resource allocation instruction, so as to guarantee the supply of high-priority task computing resources. The genetic algorithm optimization scheduling module generates chromosome coding through parallel instances, the coding structure maps the processor resource allocation ratio and the task sequence; the historical scene data simulates the actual load pressure, the fitness function adjusts the task maximum delay and resource utilization weight coefficient according to the real-time system state, and realizes the dynamic optimization of the resource allocation strategy.

[0091] The output synchronization control module uses a hardware overlay to directly synthesize radar range resolution data and vehicle profile data, avoiding software rendering delays. A vertical synchronization signal triggers a direct memory access transmission mechanism, bypassing CPU intervention and compressing the output delay of the video fusion result. Output delay feedback forms a closed-loop optimization link. The fault tolerance module monitors and identifies algorithm deadlocks through iterative cycles. A three-level hardware reset command clears abnormal states in the computing module, and a pre-stored downgrade scheduling strategy maintains basic functional operations.

[0092] Sensor loss events trigger a task invalidation marking mechanism. The genetic algorithm module updates the fitness function to mask invalid task weights, and the computation module reclaims processing resources and reallocates them to valid tasks. The video fusion task switches between high-precision registration and fast alignment algorithms based on the distance range boundary. In low-light environments, resolution downsampling is used to balance computational load and processing accuracy. This technical solution mitigates the real-time degradation caused by high-concurrency video processing through hierarchical task scheduling, dynamic resource optimization, and hardware acceleration.

[0093] In the present invention, AHD (Analog High Definition) refers to analog high-definition signal, which is an analog standard for transmitting high-definition video via coaxial cable. It supports resolutions up to 720p or 1080i and is used for vehicle external cameras to capture real-time environmental video. It is converted into a digital signal through a dedicated decoding chip to eliminate transmission delay and noise interference.

[0094] BT.656 is a digital video interface standard developed by the International Telecommunication Union (ITU-R). It defines a parallel data transmission protocol, including vertical synchronization (VSYNC), horizontal synchronization (HSYNC), and pixel clock signals. In this invention, it is used as the output format after AHD signal conversion to ensure that the video stream is transmitted in YCbCr 4:2:2 format with a stable frame rate of 25-30fps, providing a digital source with unified timing for subsequent processing.

[0095] CVBS (Composite Video Blanking Sync) represents a composite video blanking synchronization signal, which combines brightness, chrominance, blanking pulse, and synchronization signals. It is transmitted via a single-channel coaxial cable. In this application, it specifically refers to low-light environment video captured by an infrared camera. After being unpacked using the USB protocol, it is restored to the original imaging data. It supports NTSC / PAL formats and has a typical horizontal resolution of 640×480.

[0096] USB (Universal Serial Bus) is a protocol that defines the electrical specification and communication architecture for data transmission between devices, including control transfer, bulk transfer and isochronous transfer modes, in this invention used for implementing protocol unpacking for CVBS infrared signals, realizing zero-copy data transfer with a bandwidth of up to 5Gbps (USB 3.0) and reducing video stream processing delay.

[0097] YUV422 is a color space encoding format in which the luminance component (Y) is fully sampled and the chrominance components (U and V) are half-sampled horizontally (4:2:2 ratio), with each pixel occupying 16 bits of storage. In this invention, it is used as the output format after infrared video unpacking, balancing data compression and visual fidelity, reducing memory usage by 30% with chrominance sampling, while preserving key edge features.

[0098] DMA (Direct Memory Access) refers to a direct memory access technology that enables direct data transmission between memory and peripherals through a dedicated hardware controller, bypassing CPU intervention. In this invention, it is used for the physical transmission channel of video frames, with a transmission bandwidth of up to 4GB / s under the AXI bus architecture and a delay of less than 1μs, significantly reducing the output delay from video memory to the HDMI interface.

[0099] HDMI (High-Definition Multimedia Interface) is a high-definition multimedia interface that supports uncompressed audio and video stream transmission, using TMDS encoding protocol with a maximum bandwidth of 18Gbps (HDMI 2.0). In this invention, it is used to connect display devices to output fusion video, compatible with 1080p@60Hz resolution, and realize control synchronization between devices through CEC protocol.

[0100] Lux is a unit of illumination intensity, 1 lux is equal to 1 lumen per square meter. In this invention, the environmental light threshold (50 lux) is quantified and detected in real time based on a photosensitive sensor, triggering video down-sampling processing. 50 lux corresponds to a dusk or low-light environment, and below this value, the low-illumination optimization mode is started.

Claims

1. A driving assistance device based on video image processing, characterized in that: include: Multi-channel video acquisition module, synchronously receives multiple external video signals, performs standardized conversion on the video signals, and outputs a frame buffer queue with time stamp alignment; The hierarchical computing task decoupling module receives the frame buffer queue of the multi-channel video acquisition module, identifies the task type of the video data in the frame buffer queue, and marks the tasks with priority tags according to preset real-time rules. The priority tags include: marking the radar data parsing and distance superposition tasks as the first priority tag; marking the vehicle outline generation task as the second priority tag; marking the video fusion task as the third priority tag; and outputting the data stream with priority tags; a computing module configured to receive the data stream with priority tags output by the hierarchical computing task decoupling module and the resource allocation instruction sent by the genetic algorithm optimization scheduling module; And configured as: Processing the data stream corresponding to the first priority tag to generate a first priority task processing result; Processing the data stream corresponding to the second priority tag to generate a second priority task processing result; Processing the data stream corresponding to the third priority tag to generate a third priority task processing result; The genetic algorithm optimization scheduling module receives the priority-labeled data stream from the hierarchical computing task decoupling module and runs multiple sets of genetic algorithm instances in parallel to generate resource allocation strategies. The genetic algorithm instances include chromosome codes and fitness functions. The chromosome codes represent the processing resource allocation ratio and task sequence. The module loads historical scene data to simulate the video processing task flow and evaluates the optimization effect of the resource allocation strategy on task delay. The module selects the resource allocation strategy with the highest fitness and sends the resource allocation instructions generated based on the resource allocation strategy to the computing module. The resource allocation instructions are used to control the computing module to execute task processing according to the resource allocation strategy. The output synchronization control module receives the first priority task processing result and the second priority task processing result output by the calculation module, and receives the third priority task processing result output by the calculation module; the output synchronization control module synthesizes the radar distance resolution data in the first priority task processing result and the vehicle outline data in the second priority task processing result through the hardware overlay layer; performs a frame rate synchronization operation on the third priority task processing result, and outputs the synchronization processing result to the display device.

2. The driving assistance device based on video image processing according to claim 1, characterized in that: The multi-channel video acquisition module is configured as follows: Receive analog high-definition signals, perform signal conversion processing and output digital video stream; Receive composite video blanking synchronization infrared signal, implement data depacketization and output YUV color coding format data through universal serial bus protocol; Receive digital video stream and YUV color encoding format data, use double buffering mechanism to cache input frames, and store the next frame in an independent storage area while the current frame is being processed; Perform timestamp calibration processing on the video stream data in the independent storage area, and use an external clock source to calibrate the video stream timestamps corresponding to the digital video stream and the YUV color encoding format data to microsecond accuracy to form a frame buffer queue.

3. The driving assistance device based on video image processing according to claim 2, characterized in that: The hierarchical computing task decoupling module is configured as follows: Receive video data from the frame buffer queue and parse the metadata of the video data to obtain resolution parameters and signal source type; Based on the analysis results of resolution parameters and signal source type, the video data is classified into radar data stream, rear-view video stream and front-view fusion video stream; A first priority label is injected into the classified radar data stream, a second priority label is injected into the classified rear-view video stream, and a third priority label is injected into the classified front-view fusion video stream to form a data stream with priority labels.

4. The driving assistance device based on video image processing according to claim 3, characterized in that: The genetic algorithm optimization scheduling module is configured as follows: Load the preset historical scene dataset and generate the concurrent peak load of multiple video channels based on the historical scene dataset; Inject the data stream with priority tags into the preset virtualization environment and apply resource allocation policy to process the data stream with priority tags; Record the completion delays of the first-priority task, the second-priority task, and the third-priority task under the resource allocation strategy; The recorded completion delay is input into the fitness function to calculate the evaluation value. The fitness function variables include the maximum task delay and resource utilization. The function weight coefficient is dynamically adjusted according to the processor load rate and memory usage.

5. The driving assistance device based on video image processing according to claim 4, characterized in that: The genetic algorithm optimization scheduling module is also configured to: In response to a received vehicle gear signal or bus command; When the vehicle gear signal or bus instruction indicates a reverse mode, an instruction is sent to the computing module to suspend processing of the data stream corresponding to the third priority task and reallocate the released computing resources to the first priority task and the second priority task; When the vehicle gear signal or bus instruction indicates a forced radar mode, an instruction is sent to the computing module to enable the first priority task to exclusively occupy the processing resource ratio defined by the resource allocation strategy; The resource allocation strategy is a resource allocation strategy when the fitness evaluation value reaches a preset threshold.

6. The driving assistance device based on video image processing according to claim 5, characterized in that: The output synchronization control module is configured as follows: Receive radar distance resolution data from the first priority task processing result, call a preset video processing engine to write the radar distance resolution data into a hardware overlay layer of a display controller in pixel format; receiving a processing result of a third priority task, and storing the processing result of the third priority task in a video memory; When the vertical synchronization signal of the display device is at the starting point of the vertical blanking period, triggering the physical transmission channel of the direct memory access controller to transmit the processing result of the third priority task in the video memory; The actual image output delay of the high-definition multimedia interface is monitored and fed back to the fitness function weight adjustment parameter of the genetic algorithm optimization scheduling module.

7. The driving assistance device based on video image processing according to claim 6, characterized in that: Also includes: The hierarchical computing task decoupling module receives the sensor missing signal and marks the task corresponding to the missing sensor as invalid; The genetic algorithm optimization scheduling module receives the invalid state mark and updates the fitness function to shield the weight parameters of the tasks corresponding to the invalid state mark; The resource allocation strategy is regenerated based on the updated fitness function, and instructions are sent to the computing module to reclaim the processing resources of the tasks corresponding to the invalid status mark, and the recovered resources are allocated to the priority task processing flow corresponding to the valid tasks. The priority task processing flow includes the first priority task processing flow, the second priority task processing flow and the third priority task processing flow.

8. The driving assistance device based on video image processing according to claim 7, characterized in that: The genetic algorithm optimization scheduling module is also configured to: Obtain external sensor data to determine the boundary value of the preset distance range; During the video fusion task processing, when the input data is within the preset distance range boundary value, the high-precision registration algorithm is executed; when the input data exceeds the preset distance range boundary value, the fast alignment algorithm is executed; The ambient light intensity is detected by a photosensor. When the light intensity is lower than 50 lux, the collected video data is downsampled to generate downsampled frame data, and the downsampled frame data is input into the third priority task processing flow to generate the third priority task processing result.

9. The driving assistance device based on video image processing according to claim 8, characterized in that: It also includes a fault tolerance module, which is connected to the genetic algorithm optimization scheduling module. The fault tolerance module is configured as follows: Monitor the iteration cycle of the genetic algorithm instance of the genetic algorithm optimization scheduling module and record the duration of the iteration cycle; When the iteration cycle duration exceeds the preset timeout threshold, a hardware reset instruction is sent to the computing module; The pre-stored downgraded scheduling policy is loaded to replace the current resource allocation policy, and the downgraded scheduling policy is injected into the computing module for execution.

10. An assisted driving system based on video image processing, applied to the assisted driving device based on video image processing according to any one of claims 1 to 9, characterized in that: include: A multi-channel video acquisition module is configured to synchronously receive multiple external video signals, perform standardized conversion on the video signals, and output a frame buffer queue with time stamp alignment; The hierarchical computing task decoupling module is connected to the multi-channel video acquisition module and is configured as follows: Receive the frame buffer queue, identify the task type of the video data, and mark the priority tags according to the preset real-time rules, including: marking the radar data parsing and distance superposition task as the first priority tag, marking the vehicle outline generation task as the second priority tag, marking the video fusion task as the third priority tag, and outputting the data stream with the priority tags; a computing module connected to the hierarchical computing task decoupling module, configured to: receive data flows with priority labels and resource allocation instructions, process data flows corresponding to first priority labels to generate first priority task processing results, process data flows corresponding to second priority labels to generate second priority task processing results, and process data flows corresponding to third priority labels to generate third priority task processing results; The genetic algorithm optimization scheduling module connects the hierarchical computing task decoupling module and the computing module and is configured to: receive priority-labeled data streams, run multiple groups of genetic algorithm instances in parallel to generate resource allocation strategies, load historical scene data to simulate video processing task flows, and send resource allocation instructions generated based on the resource allocation strategies to the computing module. The resource allocation instructions are used to control the computing module to execute task processing according to the resource allocation strategies. an output synchronization control module connected to the calculation module and configured to: receive the first priority task processing result, the second priority task processing result, and the third priority task processing result output by the calculation module, synthesize the radar distance resolution data and the vehicle profile line data through a hardware overlay, perform a frame rate synchronization operation on the third priority task processing result, and output the synchronized processing result to a display device; The fault-tolerant module connects the genetic algorithm optimization scheduling module and the computing module, and is configured to: monitor the iteration cycle length of the genetic algorithm instance. When the iteration cycle length exceeds the preset timeout threshold, it sends a hardware reset instruction to the computing module and injects a degraded scheduling strategy into the computing module to replace the current resource allocation strategy.

Images