Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment, and the method comprises the steps: obtaining a continuous frame image, carrying out the motion detection of the continuous frame image, obtaining each motion region in the continuous frame image and the speed of each motion region, determining the target type of each motion region through semantic recognition, determining a priority corresponding to each motion area according to the speed and the target type of each motion area, and performing perception recognition on the motion areas with different priorities by adopting corresponding perception strategies to obtain a perception result; according to the method, perception identification is focused on a dynamic area and differentiated processing is carried out, a large amount of redundant calculation in a static or non-key area is effectively avoided, system calculation resources are saved, hardware loads are reduced, key targets can be ensured to be preferential, the real-time performance and response speed of the system are improved, and vehicle driving safety is ensured.

Description

Technical Field

[0001] This application relates to the field of vehicle image processing technology, and in particular to an image processing method, apparatus and electronic device. Background Technology

[0002] With the rapid development of automotive driver assistance technology, in-vehicle cameras have become a standard feature of advanced driver assistance systems (ADAS). Traditional visual processing algorithms process the full-frame image input from the in-vehicle camera at a fixed frame rate. However, in driving scenarios such as highways or parking lots, visual content often remains relatively stable for extended periods or only changes in localized areas. Global image processing methods result in a large amount of redundant computation when the scene remains largely unchanged, leading to a waste of computing resources. Furthermore, high-frame-rate global processing of the camera can cause excessive hardware load, making it difficult to meet the system's real-time response requirements.

[0003] Event cameras such as dynamic vision sensors trigger data output through pixel-level brightness changes. Although they only generate asynchronous event streams in the dynamic area of ​​the camera's field of view, their hardware costs are high, and they are incompatible with the signal format of traditional RGB (Red Green Blue, the three primary colors) cameras. They require additional dedicated processing chips, increasing deployment costs, and are difficult to integrate with existing automotive systems. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and electronic device to solve the technical problems that global image processing methods easily lead to waste of computing resources and excessive hardware load, while event cameras are costly and incompatible.

[0005] This application provides an image processing method, the method comprising: acquiring consecutive frame images; performing motion detection on the consecutive frame images to obtain each motion region in the consecutive frame images and the velocity of each motion region; determining the target type of each motion region through semantic recognition, and determining the priority of each motion region according to the velocity and target type of each motion region; and performing perception recognition on motion regions of different priorities using corresponding perception strategies to obtain perception results.

[0006] In one embodiment of this application, motion detection of the continuous frame images includes: performing inter-frame difference operations on the continuous frame images to determine each first initial motion region in the continuous frame images; performing optical flow operations on the continuous frame images to determine each second initial motion region in the continuous frame images and the velocity of each second initial motion region; and combining the velocities of each first initial motion region, each second initial motion region, and each second initial motion region to obtain each motion region and the velocity of each motion region.

[0007] In one embodiment of this application, performing inter-frame difference operations on the continuous frame images to determine each first initial motion region in the continuous frame images includes: calculating the pixel grayscale value and average brightness of each frame in the continuous frame images; calculating the inter-frame pixel grayscale value difference based on the pixel grayscale values ​​of two adjacent frames to generate an inter-frame difference image, and performing a weighted calculation based on the average brightness of each frame to obtain an inter-frame difference threshold; binarizing the inter-frame difference image based on the inter-frame difference threshold to obtain a binarization mask, and marking the continuous frame images based on the binarization mask to obtain each first initial motion region.

[0008] In one embodiment of this application, optical flow calculation is performed on the continuous frame images to determine each second initial motion region and the velocity of each second initial motion region in the continuous frame images. This includes: selecting pixels from the continuous frame images according to a preset window, and constructing an overdetermined system of equations based on the assumption of constant brightness, constrained by the fact that all pixels within the window have the same optical flow; obtaining the optical flow of each local region by solving the overdetermined system of equations; comparing a preset velocity threshold with the velocity of each local region; filtering each local region according to the comparison result to obtain each second initial motion region and the velocity of each second initial motion region, wherein the velocity of the local region is obtained based on the optical flow of the local region.

[0009] In one embodiment of this application, the target type of each motion region is determined by semantic recognition, and the priority of each motion region is determined according to the speed and target type of each motion region. This includes: performing semantic recognition on each motion region through a semantic model to obtain the target type of each motion region, wherein the target type has a corresponding confidence level; performing a weighted calculation on the speed of each motion region and the confidence level of the corresponding target type to obtain a priority score for each motion region; comparing the priority score of each motion region with a preset score range for each priority, and determining the priority of each motion region according to the comparison result.

[0010] In one embodiment of this application, a weighted calculation is performed on the speed of each motion region and the confidence level of the corresponding target type, including: determining a speed threshold corresponding to each motion region according to the current speed scenario and the target type of each motion region, wherein the current speed scenario includes a high-speed scenario or a low-speed scenario; normalizing the speed of each motion region based on the speed threshold corresponding to each motion region to obtain a normalized speed of each motion region; calculating a priority score of each motion region based on the normalized speed of each motion region, a preset speed weight, the confidence level of the target type corresponding to each motion region, and a semantic weight, wherein the semantic weight of the target type is determined based on the false negative risk level of the target type, and the higher the false negative risk level of the target type, the greater the semantic weight of the target type.

[0011] In one embodiment of this application, a corresponding perception strategy is used to perceive and identify motion regions with different priorities, including: when the priority of the motion region is high, allocating corresponding computing power according to the preset weight of the high priority and the area of ​​the motion region, and perceiving and identifying the motion region through a full model to obtain the current perception result of the motion region; when the priority of the motion region is medium, allocating corresponding computing power according to the preset weight of the medium priority and the area of ​​the motion region, and perceiving and identifying the motion region through a lightweight model to obtain the current perception result of the motion region; when the priority of the motion region is low, allocating corresponding computing power according to the preset weight of the low priority and the area of ​​the motion region, and predicting the current perception result of the motion region based on the historical perception results of the motion region.

[0012] In one embodiment of this application, predicting the current perception result of the motion region based on the historical perception result of the motion region includes: determining the acceleration of the motion region based on the historical perception result of the motion region; predicting the current perception result of the motion region based on the historical perception result of the motion region using a long-term prediction model when the acceleration of the motion region is greater than a preset acceleration threshold; and predicting the current perception result of the motion region based on the historical perception result of the motion region using a short-term prediction model when the acceleration of the motion region is less than or equal to the preset acceleration threshold.

[0013] This application also provides an image processing apparatus, the apparatus comprising: an image acquisition module for acquiring consecutive frame images; a motion detection module for performing motion detection on the consecutive frame images to obtain each motion region in the consecutive frame images and the velocity of each motion region; a semantic recognition module for determining the target type of each motion region through semantic recognition; an information processing module for determining the priority of each motion region according to the velocity and target type of each motion region; and a perception recognition module for performing perception recognition on motion regions of different priorities using corresponding perception strategies to obtain perception results.

[0014] This application also provides an electronic device, the electronic device comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method as described above.

[0015] The beneficial effects of this application are as follows: The image processing method, apparatus, and electronic device proposed in this application acquire moving regions, their speeds, and target types in continuous frame images through motion detection and semantic recognition. Based on the speed and target type of the moving regions, the priority of each region is determined, and a corresponding perception strategy is adopted to perceive and identify the moving regions according to their priority. This method can focus on dynamic regions and perform differentiated processing, effectively avoiding a large amount of redundant computation in static or non-critical regions. It saves system computing resources and reduces hardware load, while ensuring the priority of critical targets, thereby improving system real-time performance and response speed, and ensuring vehicle driving safety. Furthermore, this method can be directly integrated into existing vehicle cameras and vehicle systems without hardware upgrades or additional chip deployment, offering strong compatibility and low cost. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] In the attached diagram: Figure 1 A schematic diagram illustrating the implementation environment of an image processing method provided in an embodiment of this application; Figure 2 A flowchart illustrating an image processing method provided in one embodiment of this application; Figure 3 This is a flowchart of motion detection provided in an embodiment of this application; Figure 4 This is a block diagram of an image processing apparatus provided in one embodiment of this application; Figure 5 This is an architecture diagram of an image processing system provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer system for an electronic device provided in one embodiment of this application. Detailed Implementation

[0018] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0019] The illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0021] The embodiments of this application respectively provide an image processing method, an image processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail below.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation environment of an image processing method provided in an embodiment of this application, such as... Figure 1As shown, the implementation environment may include an intelligent driving vehicle 110 and a computer device 120, wherein the intelligent driving vehicle 110 is equipped with an onboard camera. The computer device 120 may be at least one of a GPU (Graphics Processing Unit) computer, a GPU computing cluster, a microcomputer, an embedded computer, a neural network computer, etc., without limitation. The computer device 120 may be configured within the intelligent driving vehicle 110, or it may be an electronic device independent of the intelligent driving vehicle 110, without limitation. Continuous frame images can be acquired through the onboard camera of the intelligent driving vehicle 110 and provided to the computer device 120 for image processing.

[0023] In a schematic manner, computer device 120 acquires continuous frame images; performs motion detection on the continuous frame images to obtain each moving region and its speed; determines the target type of each moving region through semantic recognition, and determines the priority of each moving region based on its speed and target type; and applies corresponding perception strategies to moving regions of different priorities for perception and recognition, obtaining perception results. It can be seen that the technical solution of this application acquires moving regions, their speeds, and target types in continuous frame images through motion detection and semantic recognition, determines the priority of moving regions based on their speed and target type, and applies corresponding perception strategies to perceive and recognize moving regions according to their priority. This allows for focusing on dynamic regions and performing differentiated processing, effectively avoiding a large amount of redundant calculation in static or non-critical regions. This saves system computing resources and reduces hardware load, while ensuring priority for critical targets, thereby improving system real-time performance and response speed, and ensuring vehicle driving safety. Furthermore, this solution can be directly integrated into existing vehicle cameras and vehicle systems without hardware upgrades or additional chip deployment, offering strong compatibility and low cost.

[0024] The image processing method provided in this application embodiment can be specifically executed by computer device 120, and correspondingly, the image processing device can be disposed in computer device 120.

[0025] Please see Figure 2 , Figure 2 This is a flowchart illustrating an image processing method provided in one embodiment of the present application. This image processing method can be applied to... Figure 1The implementation environment is shown, and the image processing method is specifically executed by computer device 120 within that implementation environment. It should be understood that this image processing method can also be applied to the motion scenarios of embodied intelligent robots and specifically executed by computer devices within the embodied intelligent robot. This image processing method can also be applied to other exemplary implementation environments and specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which this image processing method is applicable. Figure 2 As shown, in an exemplary embodiment, the image processing method includes at least steps S210 to S240, which are described in detail below: Step S210: Obtain consecutive frame images.

[0026] Step S220: Perform motion detection on the continuous frame images to obtain each motion region and the velocity of each motion region in the continuous frame images.

[0027] Step S230: Determine the target type of each motion region through semantic recognition, and determine the priority of each motion region according to the speed and target type of each motion region.

[0028] Step S240: Use corresponding perception strategies to perceive and identify motion regions of different priorities to obtain perception results.

[0029] In step S210, a continuous frame image refers to images at multiple consecutive moments, which can be an image stream or video slice arranged in chronological order. The interval between moments can be fixed or variable. Therefore, a continuous frame image includes multiple consecutive image frames, or a continuous frame image includes multiple consecutive frames.

[0030] In some embodiments, a series of static images can be captured sequentially at fixed or variable intervals at continuous time points using image acquisition devices installed on the vehicle, such as vehicle-mounted cameras or cameras, as continuous frame images.

[0031] In step S220, the motion region refers to the region consisting of a set of pixels whose position, shape, or size changes between two or more consecutive frames. The velocity of the motion region refers to the amount of displacement change of the motion region per unit time, including size (i.e., rate) and direction, indicating how many pixels the motion region has moved from the previous frame to the current frame and in which direction. Since a continuous frame image comprises multiple consecutive frames, each motion region in the continuous frame image refers to the motion region of each frame of the continuous frame image, and the number of motion regions in each frame can be one or more. At least one of the following algorithms can be used to perform motion detection on the continuous frame image to determine the motion regions and their velocities.

[0032] In some embodiments, motion detection of consecutive frame images includes performing motion detection on consecutive frame images using a first preset algorithm and a second preset algorithm respectively, and obtaining a first motion detection result and a second motion detection result respectively, wherein the first preset algorithm and the second preset algorithm are different, or the first preset algorithm and the second preset algorithm are the same; performing an AND operation on the first motion detection result and the second motion detection result to obtain a final motion detection result, including each motion region and the velocity of each motion region.

[0033] When the first and second preset algorithms are the same, the reliability of motion detection results can be improved through secondary verification. When the first and second preset algorithms are different, the advantages of different algorithms can be combined to compensate for their shortcomings, cover more complex motion scenarios, and improve adaptability to different dynamic scenarios.

[0034] In step S230, the target type of the motion area refers to the category of the target object (hereinafter referred to as the target) represented by the motion area, such as vehicles, pedestrians, animals, traffic lights, buildings, sky, etc. The priority of the motion area represents the degree of influence of the target object represented by the motion area on the current driving safety of the vehicle. The higher the priority of the motion area, the greater its influence on the current driving safety of the vehicle.

[0035] By determining the priority of a movement area based on its speed and target type, the system can prioritize the perception and identification of high-risk targets, such as vehicles approaching at high speeds or pedestrians suddenly appearing, thus shortening the response time to critical threats. It also allows limited computing power to be concentrated on high-priority targets for deep perception, while low-priority targets, such as vehicles moving away at low speeds, are handled with simplified processing, avoiding resource waste. Furthermore, by combining physical motion with semantic understanding, the solution in this embodiment is more consistent with human driving logic; for example, it prioritizes pedestrians crossing laterally rather than cars moving slowly in the same direction.

[0036] In step S240, the perception results of the moving region include velocity information, position information, category information, width and height information, and feature map information of the moving region at each frame or time. Different priority perception strategies are allocated different computing resources, with higher priority perception strategies receiving more computing resources than medium priority strategies, which in turn receive more computing resources than low priority strategies. For example, higher priority perception strategies may have higher processing frame rates, while lower priority strategies may have lower processing frame rates; or, higher priority perception strategies may have more computing power allocated to them, while lower priority strategies may have less computing power allocated to them; or, higher priority perception strategies may use more accurate detection models, while lower priority perception strategies may use lighter detection models; or, higher priority moving regions may be prioritized for perception and recognition.

[0037] In some embodiments, the high-priority perception strategy includes: allocating 70% of computing resources to high-priority motion areas; the medium-priority perception strategy includes: allocating 25% of computing resources to medium-priority motion areas; and the low-priority perception strategy includes: allocating 5% of computing resources to low-priority motion areas.

[0038] By allocating different computing resources to different motion areas and performing differentiated perception and recognition, a large amount of redundant computing in non-critical areas can be effectively avoided, thereby saving computing resources and reducing hardware load, and ensuring priority for key targets such as pedestrians and vehicles.

[0039] In one embodiment of this application, step S220 includes: performing inter-frame difference operation on consecutive frame images to determine each first initial motion region in the consecutive frame images; performing optical flow operation on consecutive frame images to determine each second initial motion region and the velocity of each second initial motion region in the consecutive frame images; and combining each first initial motion region, each second initial motion region and the velocity of each second initial motion region to obtain each motion region and the velocity of each motion region.

[0040] In this embodiment, motion detection is performed on consecutive frame images using a first preset algorithm and a second preset algorithm, respectively. This includes performing inter-frame difference operations on the consecutive frame images using an inter-frame difference algorithm to obtain a first motion detection result, including each first initial motion region, wherein the first preset algorithm is an inter-frame difference algorithm; and performing optical flow operations on the consecutive frame images using an optical flow algorithm to obtain a second motion detection result, including each second initial motion region and the velocity of each second initial motion region, wherein the second preset algorithm is an optical flow algorithm. Inter-frame difference operations can quickly determine motion regions, improving the efficiency of motion detection. Optical flow operations can compensate for the deficiency of inter-frame difference operations in obtaining velocity information. Performing motion detection twice using inter-frame difference and optical flow methods achieves a balance between accuracy and efficiency, avoiding reliance on high-overhead models.

[0041] When performing inter-frame difference operations on consecutive frame images, based on the assumption that "the gray value of pixels in static areas changes little while that in moving areas changes much", a binary mask can be generated by calculating the difference in pixel gray values ​​between consecutive frame images, marking the potential moving areas in the consecutive frame images as the first initial moving areas.

[0042] In one embodiment of this application, inter-frame difference operations are performed on consecutive frame images to determine each first initial motion region in the consecutive frame images, including: calculating the pixel grayscale value and average brightness of each frame in the consecutive frame images; calculating the difference in pixel grayscale values ​​between adjacent frames based on the pixel grayscale values ​​of two adjacent frames to generate an inter-frame difference image, and performing a weighted calculation based on the average brightness of each frame to obtain an inter-frame difference threshold; binarizing the inter-frame difference image based on the inter-frame difference threshold to obtain a binarization mask, and marking the consecutive frame images based on the binarization mask to obtain each first initial motion region.

[0043] This embodiment uses adaptive thresholding for binarization, which reduces interference when there are strong changes in lighting conditions and improves system robustness.

[0044] Please see Figure 3 , Figure 3 This is a flowchart of motion detection provided in an embodiment of this application, such as... Figure 3 As shown, a frame sequence is a series of consecutive frame images, in which each frame image can constitute at least one pair of adjacent frames. For each pair of adjacent frames, the gray value difference of the pixel at the same position in the two adjacent frames is calculated, which is the inter-frame pixel gray value difference. An inter-frame difference image (referred to as a difference map) is generated based on the pixel gray value difference of pixels at all positions, and a weighted calculation is performed based at least on the average brightness of the current frame in the two adjacent frames to obtain the inter-frame difference threshold (i.e., adaptive threshold) corresponding to the inter-frame difference image. The inter-frame difference image is then binarized according to the inter-frame difference threshold corresponding to each inter-frame difference image to obtain the corresponding binarization mask.

[0045] The binarization method is as follows: Equation (1) in, M ( x , y ) is a binary mask, | I t ( x , y )- I t-1 ( x , y | represents the pixels in two adjacent frames ( x , y The difference in grayscale values ​​between pixels, that is, the grayscale values ​​between pixels in different frames. I t ( x , y ) is the pixel in the current frame ( x , y The grayscale value of ) I t-1 ( x ,y ) is the pixel in the previous frame ( x , y The grayscale value of ) x , y () represents the coordinates of a pixel. τ dynamic The threshold is adaptive. In equation (1), 0 represents black and 1 represents white. When the difference in grayscale value between pixels in a certain frame is greater than the inter-frame difference threshold, the pixel value of that pixel is set to 1 (white), and vice versa. Two adjacent frames refer to two adjacent image frames in a continuous frame image. The previous frame is the first frame in the two adjacent frames, and the current frame is the second frame in the two adjacent frames. The timestamp of the previous frame is earlier than the timestamp of the current frame.

[0046] In some embodiments, when calculating the inter-frame difference threshold corresponding to each inter-frame difference image, the inter-frame difference threshold can be obtained by weighting the average brightness (i.e., average grayscale value) of the current frame in the two adjacent frames corresponding to the inter-frame difference image. The calculation method is as follows: τ dynamic = k · u ( I t )+ c Equation (2) in, τ dynamic This is the inter-frame difference threshold. u ( I t The value represents the average brightness of the current frame. k This is a scaling factor used to control the weight of the average brightness on the inter-frame difference threshold. c This is a noise margin used to filter out tiny pixel noise.

[0047] In other embodiments, when calculating the inter-frame difference threshold corresponding to each inter-frame difference image, to prevent false triggering caused by sudden changes in illumination, the inter-frame difference threshold can be obtained by weighting the average brightness of the current frame and the average brightness of the previous frame in the two adjacent frames corresponding to the inter-frame difference image. The calculation method is as follows: Equation (3) in, τ dynamic This is the inter-frame difference threshold. u ( I t The value represents the average brightness of the current frame. u ( I t-1 () represents the average brightness of the previous frame. k This is the proportionality coefficient.c This is for noise tolerance.

[0048] For example, k It can be preset to 0.1. c It can be preset to 5.

[0049] In some embodiments, image data of urban roads, highways, and other scenes can be collected in advance and compared with the scaling factor. k and noise margin c Optimization is performed. For urban road scenarios, image data of peak traffic hours, normal traffic flow, and backlit scenes can be collected, with peak traffic hours including morning and / or evening peak hours. For highway scenarios, tunnel image data, normal vehicle image data, and backlit scenes can be collected, with tunnel image data including at least one of the following: tunnel entry, tunnel exit, and tunnel interior images. For other scenarios, garage image data, rainy day image data, and rural road image data can be collected, with garage image data including at least one of the following: garage entry, garage exit, and garage interior images. The collected image data for each scenario is labeled to form a labeled dataset. Based on this labeled dataset, a grid search method is used to optimize the scaling factor. k and noise margin c Optimize.

[0050] When performing optical flow calculations on consecutive frame images, based on the assumptions of "constant pixel brightness" and "consistent motion in local regions", the second initial motion region can be accurately located and the motion velocity can be obtained by calculating the displacement vector (optical flow) of pixels between adjacent frames in consecutive frame images. This can make up for the deficiency of inter-frame difference algorithms in obtaining velocity information.

[0051] In one embodiment of this application, optical flow calculation is performed on consecutive frame images to determine each second initial motion region and the velocity of each second initial motion region in the consecutive frame images. This includes: selecting pixels from the consecutive frame images according to a preset window, and constructing an overdetermined system of equations based on the assumption of constant brightness, constrained by the fact that all pixels within the window have the same optical flow; obtaining the optical flow of each local region by solving the overdetermined system of equations; comparing a preset velocity threshold with the velocity of each local region; filtering each local region according to the comparison result to obtain each second initial motion region and the velocity of each second initial motion region, wherein the velocity of the local region is obtained based on the optical flow of the local region.

[0052] Compared to calculating optical flow independently for each pixel, this embodiment solves for optical flow using information from multiple pixels within a window. This avoids the impact of image noise or minute brightness variations and enables lightweight computation, reducing computational resources.

[0053] Please continue reading Figure 3 ,like Figure 3 As shown, the velocity of feature points can be tracked and the moving region determined through sparse optical flow calculation. Assuming a pixel ( x , y )exist t The brightness (i.e., grayscale value) at time is I ( x , y , t ), t +Δ t At what moment should the brightness shift to ( x + u Δ t , y + v Δ t Based on the assumption of constant brightness, the following equation can be obtained: I ( x + u Δ t , y + v Δ t , t +Δ t )= I ( x , y , t Equation (4) in, I ( x + u Δ t , y + v Δ t , t +Δ t ) is pixels ( x + u Δ t , y + v Δ t )exist t +Δ t Brightness at any moment I ( x , y , t ) is pixels ( x , y )exist t Brightness at any moment u = dx / dt for x Optical flow velocity in the direction, v = dy / dt for y Optical flow velocity in the direction, Δ t This represents the change over time.

[0054] Performing a Taylor expansion on the left side of equation (4) and neglecting higher-order terms, we obtain the fundamental equation of optical flow, as follows: I x u + I y v + I t =0 (5) in, I x = For image x Spatial gradient of direction, I y = For image y Spatial gradient of direction, I t = This represents the time gradient.

[0055] The preset window size can be 3x3. The Lucas-Kanad method can be used to solve the local optical flow, selecting a pixel-based approach. x , y Taking a 3x3 window centered at (), and assuming that all pixels within the window move at the same speed (i.e., all pixels have the same optical flow), we construct an overdetermined system of equations, as follows: Equation (6) in, I x1 ~ I x9 Each pixel within the window x Directional light flow I y1 ~ I y9 Each pixel within the window y Directional light flow I t1 ~ I t9 These represent the optical flow of each pixel within the window in the time dimension.

[0056] The overdetermined system of equations was solved using the least squares method, and the results are as follows: Equation (7) in, u for xOptical flow velocity in the direction, v for y Optical flow velocity in the direction, A The spatial gradient matrix of the pixels within the window. A T for A The transpose of the matrix, b This is the time gradient vector.

[0057] The solution results for each window are used as the optical flow for each local region. Based on the optical flow of each local region, the optical flow velocity amplitude, i.e., the velocity of each local region, is calculated. This local region is then filtered according to the optical flow velocity amplitude. The calculation method for the optical flow velocity amplitude is as follows: Equation (8) in, v mag The amplitude of optical flow velocity. u for x Optical flow velocity in the direction, v for y The speed of light flow in the direction.

[0058] In some embodiments, filtering local regions based on optical flow velocity amplitude includes: filtering local regions with optical flow velocity amplitude greater than a preset velocity threshold as initial local regions; smoothing the optical flow velocity amplitude of the initial local regions using a sliding window across multiple frames to achieve noise filtering and obtain a smoothed optical flow velocity of the initial local regions; filtering the initial local regions with smoothed optical flow velocity greater than a preset velocity threshold as second initial motion regions, and using the corresponding smoothed optical flow velocity as the velocity of the second initial motion region.

[0059] In other embodiments, each local region is filtered based on the optical flow velocity amplitude, including: smoothing the optical flow velocity amplitude of the local region through a sliding window using multiple frames of optical flow trajectory to achieve noise filtering and obtain the smoothed optical flow velocity of the local region; filtering local regions with smoothed optical flow velocities greater than a preset velocity threshold as second initial motion regions, and using the corresponding smoothed optical flow velocity as the velocity of the second initial motion region.

[0060] For example, the smoothing method for the optical flow velocity amplitude is as follows: Equation (9) in, v filtered To smooth the optical flow velocity, v mag,ti The amplitude of optical flow velocity. N The slider window size can be set to 3 by default, or adjusted to 2-5 depending on real-time requirements.

[0061] For example, a preset speed threshold τ flow It can be 2 pixels per frame, or other values; there are no restrictions here.

[0062] like Figure 3 As shown, by fusing the difference and optical flow results, a final ROI (Region of Interest) mask is generated as the motion region.

[0063] In some embodiments, after obtaining the difference results (i.e., each first initial motion region) and the optical flow results (i.e., each second initial motion region), an AND operation can be performed on the first and second initial motion regions of the same frame to obtain each motion region in each frame and the velocity of the motion region. The AND operation is performed as follows: M final ( x , y )= M diff ( x , y )&( v filtered ( x , y > τ flow Formula (10) in, M final ( x , y () indicates the area of ​​motion. M diff ( x , y ) represents the first initial motion region, ( v filtered ( x , y > τ flow ) represents the second initial motion region.

[0064] By performing motion detection on consecutive frames using inter-frame difference and optical flow algorithms respectively, and then performing a bitwise AND operation on the motion detection results, the real-time performance of motion detection can be guaranteed and the accuracy of motion region detection can be improved.

[0065] In one embodiment of this application, the target type of each motion region is determined by semantic recognition, and the priority of each motion region is determined according to the speed and target type of each motion region. This includes: performing semantic recognition on each motion region through a semantic model to obtain the target type of each motion region, wherein the target type has a corresponding confidence level; performing a weighted calculation on the speed of each motion region and the confidence level of the corresponding target type to obtain the priority score of each motion region; comparing the priority score of each motion region with the preset score range of each priority, and determining the priority of each motion region according to the comparison result.

[0066] In this embodiment, the semantic model includes a semantic segmentation model, a semantic recognition model, or a target detection model. A lightweight semantic segmentation model, semantic recognition model, or target detection model can be used to extract target types, including pedestrians, vehicles, traffic lights, traffic signs, non-motorized vehicles, buildings, sky, and other backgrounds, and to provide corresponding confidence scores.

[0067] In some embodiments, an object detection model with an inference speed >100 FPS, such as YOLOv10-Tiny, is selected to identify various core targets in real time, thereby improving the efficiency of semantic recognition and reducing computational resources.

[0068] The core principle of priority classification is to allocate computing resources on demand. Combining the motion characteristics of the target (speed) and semantic importance (target type importance), the moving regions of the image are divided into high, medium, and low priorities. Regions with faster motion speed and higher semantic importance are allocated more computing resources to ensure that key targets, such as pedestrians and moving vehicles, are not missed and are processed first; while non-key regions, such as the sky and green belts, are allocated relatively fewer computing resources.

[0069] Therefore, different weights can be pre-assigned to different target types based on their importance, with more important target types receiving higher weights. Based on the weights assigned to the target types corresponding to a specific motion region, the speed of that motion region and the confidence level of the target type corresponding to that motion region are weighted and calculated to obtain the priority score for that motion region. This method is used to obtain the priority scores for each motion region.

[0070] This embodiment uses the speed of the moving region and the confidence level of the corresponding semantic information (target type) of the moving region for weighted calculation, achieving a balance between the confidence level of semantic information and speed, and improving the accuracy of priority division.

[0071] In one embodiment of this application, a weighted calculation is performed on the speed of each motion region and the confidence level of the corresponding target type, including: determining the speed threshold corresponding to each motion region according to the current speed scenario and the target type of each motion region, wherein the current speed scenario includes a high-speed scenario or a low-speed scenario; normalizing the speed of each motion region based on the speed threshold corresponding to each motion region to obtain the normalized speed of each motion region; calculating the priority score of each motion region based on the normalized speed of each motion region, a preset speed weight, the confidence level of the target type corresponding to each motion region, and the semantic weight, wherein the semantic weight of the target type is determined based on the false negative risk level of the target type, and the higher the false negative risk level of the target type, the greater the semantic weight of the target type.

[0072] In this embodiment, the speed threshold is not a fixed value. The same speed of a movement area has different effects on vehicle driving or the movement of an embodied intelligent robot in different speed scenarios. Therefore, by determining the speed threshold of the target type of the movement area in the current speed scenario according to the current speed scenario and the target type of the movement area, and using it as the speed threshold of the movement area, the speed of the movement area can be normalized according to the speed threshold of the movement area. This can ensure that the priority division of the movement area is consistent in different speed scenarios and improve the accuracy of priority score calculation.

[0073] In some embodiments, for vehicle driving scenarios, the current speed scenario can be determined based on current vehicle driving information such as current vehicle speed, current vehicle location information, or current navigation information. For example, if the current navigation information shows that the vehicle is currently driving on a highway, the current speed scenario is determined to be a high-speed scenario; if the current navigation information shows that the vehicle is currently driving on a city road, the current speed scenario is determined to be a low-speed scenario.

[0074] In other embodiments, for the movement scenario of the embodied intelligent robot, the current speed scenario can be determined based on the current movement speed of the embodied intelligent robot. For example, if the current movement speed of the embodied intelligent robot exceeds a preset value, the current speed scenario is determined to be a high-speed scenario; if the current movement speed of the embodied intelligent robot is less than or equal to the preset value, the current speed scenario is determined to be a low-speed scenario.

[0075] In some embodiments, determining the speed threshold corresponding to each motion region according to the current speed scene and the target type of each motion region includes: pre-calibrating the speed thresholds of targets of different target types in different speed scenes, establishing a mapping relationship between target type-speed scene-speed threshold, wherein the speed thresholds of targets of different target types in different speed scenes include the speed thresholds of targets of each target type in high-speed scenes and the speed thresholds of targets of each target type in low-speed scenes; determining the speed threshold of the target type of the motion region in the current speed scene according to the target type of the motion region and the current speed scene from the mapping relationship between target type-speed scene-speed threshold, and using it as the speed threshold corresponding to the motion region.

[0076] For example, when calibrating the speed threshold, the threshold can be dynamically adjusted by combining camera parameters, the target's preset speed, and the target's safe distance. Taking vehicle movement as an example, the cameras in the sensors of intelligent driving functions generally adopt a pinhole camera model. Therefore, the relationship between the pixel displacement of the target in the image and its actual motion can be expressed as follows: Equation (11) in, D The displacement pixels of the target in the image. v obj The actual speed of the target's movement. t frame For frame interval, f For camera focal length, d obj The distance from the target to the camera (camera). s pixel This refers to the pixel size of the camera (camera) sensor. D The unit is pixel / frame. v obj The unit is m / s. f The unit is mm. d obj The unit is m. s pixel The unit is μm. The frame interval is the reciprocal of the frame rate, which can be set to 30 FPS by default. Since automotive cameras generally use fixed-focus lenses, the camera focal length can be set to 6mm by default. The camera (camera) sensor pixel size can be set to 1.4 μm by default.

[0077] The preset speed of the target and the safe distance from the camera (camera) can be used as the parameters for different speed scenarios. v obj and d objAnd the speed threshold of each target in different speed scenarios is calibrated by Equation (11). For example, for high-speed scenarios, i.e., speeds of 60~120km / h, if the target is a vehicle, the corresponding safe distance of the target can be 100m, and the preset speed of the target can be 80km / h, which is approximately 22.2m / s. The corresponding pixel displacement is... D Since the speed threshold is approximately 5 pixels per frame, it can be set to a value greater than 5 pixels per frame. For low-speed scenes (speeds ranging from 0 to 60 km / h), if the target is a pedestrian, the safe distance to the target can be 20 meters, and the preset speed of the target can be 5 km / h, approximately 1.4 m / s, corresponding to a pixel displacement of... D Since the value is approximately 2 pixels per frame, the speed threshold can be set to a value greater than 2 pixels per frame.

[0078] Of course, the safe distance and preset speed of each target in each speed scenario can be reasonably set according to the actual situation. Here, we do not impose restrictions on the specific values ​​of the safe distance and preset speed of each target in each speed scenario.

[0079] The velocity of the motion region is normalized according to the velocity threshold corresponding to the motion region, as follows: v norm = v filtered ÷ v max Equation (12) in, v norm The normalized velocity of the motion region, v filtered The velocity of the moving area, v max This represents the velocity threshold corresponding to the motion region.

[0080] Following the above method, the velocity of each motion region is normalized to 0~1 to obtain the normalized velocity of each motion region.

[0081] The missed detection risk level of a target type is also known as the missed detection risk level of the target represented by that target type. The missed detection risk level, or missed detection cost, refers to the likelihood of a safety accident occurring if a target is missed. The higher the likelihood of a safety accident occurring if a target is missed, the higher the missed detection cost, and thus the higher the missed detection risk level. For example, the likelihood of a safety accident occurring if a pedestrian or vehicle is missed is higher, while the likelihood of a safety accident occurring if a building or the sky is missed is lower. Therefore, assigning semantic weights to target types based on the missed detection costs of different targets—with higher missed detection costs for targets, higher semantic weights—can further improve the accuracy of priority score calculation and reduce missed detection risk.

[0082] In some embodiments, target types include pedestrians, vehicles (including motor vehicles and non-motor vehicles), traffic lights, traffic signs, buildings, sky, and other backgrounds. Because missing pedestrians can easily lead to serious safety accidents (extremely high risk), missing vehicles can easily lead to safety accidents (high risk), missing traffic lights can easily affect decision-making accuracy (high risk), missing traffic signs can affect driving efficiency (low risk), buildings are static targets and generally pose no direct risk, other backgrounds are static targets and pose no direct risk, and the sky has no actual semantic value and poses no risk when missed. Therefore, the semantic weight of pedestrians > semantic weight of vehicles = semantic weight of traffic lights > semantic weight of traffic signs > semantic weight of buildings > semantic weight of other backgrounds > semantic weight of the sky.

[0083] For example, the semantic weight of a pedestrian can be 3.0, the semantic weight of a vehicle and the semantic weight of a traffic light can both be 2.5, the semantic weight of a traffic sign can be 2.0, the semantic weight of a building can be 1.0, the semantic weight of other backgrounds can be 0.5, and the semantic weight of the sky can be 0.1.

[0084] The priority score of a motion region is calculated by combining its normalized velocity, preset velocity weight, confidence level, and semantic weight for the target type. The calculation method is as follows: S priority = w v * v norm + w s * s score Equation (13) in, S priority Priority scores are assigned to the movement areas. w v For preset speed weights,v norm The normalized velocity of the motion region, w s The semantic weights for the target type corresponding to the motion region. s score The confidence level for the target type corresponding to the motion region ranges from 0 to 1. The preset velocity weight can be set to 1.0 by default, or to other values. The priority score for each motion region is calculated using the method described above.

[0085] When determining the priority of a movement area based on its priority score, a corresponding score range can be pre-set for different priorities, resulting in a preset score range for each priority. If the priority score of a certain movement area falls within the preset score range for a certain priority, then the priority corresponding to that movement area is that priority.

[0086] In some embodiments, priorities include high priority, medium priority, and low priority. The minimum value of the preset score range corresponding to high priority is greater than the maximum value of the preset score range corresponding to medium priority, and the minimum value of the preset score range corresponding to medium priority is greater than the maximum value of the preset score range corresponding to low priority. For example, the preset score range corresponding to high priority can be (1.8, +∞), the preset score range corresponding to medium priority can be (0.8, 1.8), and the preset score range corresponding to low priority can be (-∞, 0.8).

[0087] In another embodiment of this application, determining the priority of each motion region based on its speed and target type includes: determining the priority of each motion region based on its speed and the semantic weight of the corresponding target type. When the speed of a motion region is greater than 5 pixels / frame, and the semantic weight of the target type corresponding to that motion region is greater than or equal to 2.5, the priority of that motion region is determined to be high priority; when the speed of a motion region is greater than 2 pixels / frame and less than or equal to 5 pixels / frame, and the semantic weight of the target type corresponding to that motion region is greater than 1.0, the priority of that motion region is determined to be medium priority; when the speed of a motion region is less than or equal to 2 pixels / frame, and the semantic weight of the target type corresponding to that motion region is less than or equal to 1.0, the priority of that motion region is determined to be low priority.

[0088] In one embodiment of this application, a corresponding perception strategy is used to perceive and identify motion regions with different priorities, including: when the priority of the motion region is high, the motion region is perceived and identified using a full model based on a preset weight for high priority and the corresponding computing power allocated to the area of ​​the motion region, to obtain the current perception result of the motion region; when the priority of the motion region is medium, the motion region is perceived and identified using a lightweight model based on a preset weight for medium priority and the corresponding computing power allocated to the area of ​​the motion region, to obtain the current perception result of the motion region; when the priority of the motion region is low, the current perception result of the motion region is predicted based on the historical perception results of the motion region based on a preset weight for low priority and the corresponding computing power allocated to the area of ​​the motion region.

[0089] This embodiment utilizes heterogeneous computing of motion regions with different priorities within a frame, along with flexible allocation of computing resources, to further avoid wasting computing power.

[0090] For high-priority and medium-priority motion regions, the current perception result refers to the perception result obtained by perceiving and recognizing the information of the motion region in the currently processed image frame. For low-priority motion regions, the current perception result refers to the perception result obtained by predicting based on the historical perception results of the motion region. The processing frame rate for high-priority motion regions is greater than that for medium-priority motion regions, which in turn is greater than that for low-priority motion regions; the input resolution for high-priority motion regions is greater than that for medium-priority motion regions.

[0091] For example, the processing parameters for motion regions with different priorities are configured as follows: High-priority motion regions: The processing frame rate is the full frame rate, i.e., 30Hz; the detection model is the full model, such as YOLOv10x; the input resolution is the full resolution, such as 1920*1080; and computing resources are allocated according to the preset weight of high priority and the area of ​​the motion region. Medium-priority motion regions: The processing frame rate is once every 3 frames, i.e., 10Hz. The detection model is a lightweight model, such as YOLOv10N. The input resolution is downsampled, such as 960*540. Computational resources are allocated according to the preset weight of medium priority and the area of ​​the motion region. Low-priority motion regions: The processing frame rate is once every 30 frames, i.e. 1Hz. Historical feature maps from historical perception results are directly reused. Computational resources are allocated according to the preset weight of low priority and the area of ​​the motion region.

[0092] For example, the computing power allocation formula is as follows: Equation (14) in, GPU_Ratio For the first i The computing power of each motion area w i For the first i Each motion region corresponds to a preset weight for priority. S i For the first i The area percentage of the first sports area indicates the percentage of the area of ​​the first sports area. i The proportion of the pixel area of ​​each motion region in the current frame to the total image area. w j For the first j Each motion region corresponds to a preset weight for priority. S j For the first j The area percentage of each movement region. In equation (14), the numerator represents the area percentage of the first movement region. i The weighted area ratio of each motion region is represented by the denominator, which is the sum of the weighted area ratios of all motion regions in the current frame, thus achieving normalization.

[0093] The area ratio of the exercise zone is calculated as follows: S i =Motor Zone i The formula for the ratio of the number of pixels in the image to the total number of pixels is (15). in, S i For the first i The area percentage of each sports area.

[0094] By dynamically adjusting the proportion of GPU resources, the utilization rate of computing resources can be improved and the hardware load can be reduced while ensuring the accuracy of critical areas.

[0095] In some embodiments, since the cost of missing a pedestrian / vehicle is much higher than that of the background, the preset weights for high, medium and low priorities can be 3, 1 and 0.5 respectively.

[0096] Based on the priority division results, an asynchronous perception strategy is adopted, which combines frame rate differentiation, model lightweighting, resolution adaptation, and dynamic allocation of computing power. That is, high-priority motion areas consume more computing power and are processed with full frame rate and high-precision model; while low-priority motion areas consume less computing power and are processed with low frame rate and reused features. This achieves on-demand allocation of computing power, balancing real-time performance and accuracy.

[0097] In some embodiments, the perception strategy for motion regions of different priorities is adopted for perception and recognition. The method further includes: adopting a multi-threaded parallel architecture, that is, using different processing threads to perform parallel perception and recognition of motion regions of different priorities. Among them, the processing thread for high-priority motion regions has the highest priority, followed by medium and low priority, so as to avoid low-priority tasks blocking high-priority tasks and to achieve frame-level asynchronous processing.

[0098] In some embodiments, the corresponding perception strategies are used to perceive and identify motion regions of different priorities. The method further includes: data of high-priority motion regions are transferred using DMA (Direct Memory Access) to bypass the CPU (Central Processing Unit) cache and reduce data caching, while data of medium- and low-priority motion regions are transferred using ordinary memory to achieve asynchronous data transmission.

[0099] In some embodiments, the corresponding perception strategies are used to perceive and identify motion regions of different priorities. The methods also include: triggering detection once per frame for high-priority motion regions, once every 3 frames for medium-priority motion regions, and once every 30 frames for low-priority motion regions. The triggering time is synchronized with that of high-priority motion regions to avoid data conflicts.

[0100] In one embodiment of this application, predicting the current perception result of a motion region based on the historical perception result of the motion region includes: determining the acceleration of the motion region based on the historical perception result of the motion region; predicting the current perception result of the motion region based on the historical perception result of the motion region using a long-term prediction model when the acceleration of the motion region is greater than a preset acceleration threshold; and predicting the current perception result of the motion region based on the historical perception result of the motion region using a short-term prediction model when the acceleration of the motion region is less than or equal to the preset acceleration threshold.

[0101] In this embodiment, since low-priority areas are processed at low frame rates, information loss may occur. Therefore, information compensation can be performed using a temporal prediction model. By using a hybrid model of short-term and long-term prediction, and combining acceleration to dynamically switch between the temporal and long-term prediction models, both short-term and long-term predictions can be taken into account, ensuring the continuity of perception output and the integrity of information in different scenarios.

[0102] In some embodiments, the short-time prediction model is based on a linear dynamic model of the target motion, estimating the target state through a prediction-update loop, and is used for short-time prediction within 1 second for targets moving at uniform speed / uniform acceleration, such as a Kalman filter prediction model. Exemplarily, the framework of a Kalman filter prediction model includes a state vector definition, a state transition equation, an observation equation, and an update rule, wherein the state vector is defined as follows: X t =[ x t , y t , w t , h t , v xt , v yt ] T Equation (16) in, X t Let be the state vector, ( x t , y t ) represents the center coordinates of the target bounding box. w t , h t ) represents the width and height of the target bounding box, v xt , v yt ) represents the movement speed of the center coordinates of the target bounding box.

[0103] The state transition equation is expressed as follows: X t = A · x t-1 + B · u t + ω t Equation (17) in, X t For state vectors, A This is the state transition matrix based on the kinematics model, assuming the target moves at a constant velocity. B To control the input matrix, without external control, B =0, u t To control the input, ω t For process noise, ω t ~ N (0, Q ), Q Let be the covariance matrix of the state transition equation.

[0104] The observation equation is expressed as follows: Z t = H · X t + v t Equation (18) in, Z t For the observed values, H For the observation matrix, only the bounding box parameters are observed. X t For state vectors, v t To observe the noise, v t ~ N (0, R ), R Let be the covariance matrix of the observation equation.

[0105] The update rules include: low-priority motion regions are detected once every 30 frames, and the Kalman filter observations are updated with the detection results; the target state is obtained through the prediction equation in intermediate frames.

[0106] In some embodiments, the long-term prediction model learns historical motion trajectory features to achieve long-term predictions (greater than 1 second) for non-uniformly moving targets, such as pedestrians suddenly changing direction or vehicles accelerating or decelerating, thus compensating for the shortcomings of the linear assumption of the Kalman filter. Illustratively, the long-term prediction model can be an LSTM (Long Short-Term Memory) prediction model, where the input features of the LSTM prediction model include velocity information of the moving region at six historical time points, semantic weights, and priority scores, as shown below: F input =[ v x,t-5 , v y,t-5 ,…, v x,t , v y,t , w s , S priority Equation (19) in, F inputInput features to the model, ( v x,t-5 , v y,t-5 )~( v x,t , v y,t ) represents the velocity of the motion region at 6 historical moments. w s The semantic weights for the target type corresponding to the motion region. S priority Priority scores are assigned to the movement areas.

[0107] The output of the LSTM prediction model is the target state at the current time, represented as follows: F output =[ x t+1 , y t+1 , w t+1 , h t+1 , v x,t+1 , v y,t+1 Equation (20) in, F output To output a prediction for the model, ( x t+1 , y t+1 ) represents the predicted location of the moving region at the current moment. w t+1 , h t+1 ) represents the width and height of the moving area at the current moment. v x,t+1 , v y,t+1 ) represents the predicted velocity of the moving region at the current moment.

[0108] The LSTM prediction model consists of two LSTM layers with a hidden layer dimension of 64, one fully connected layer with an output dimension of 6, ReLU (Rectified Linear Unit) as the activation function, and MSE (Mean Squared Error) as the loss function.

[0109] The motion trajectory samples of pedestrians and vehicles can be selected from publicly available autonomous driving datasets such as nuScence and used as training data for LSTM prediction models. The ratio of training set to validation set can be 8:2, and the target prediction error can be less than or equal to 2 pixels. nuScence contains 1,000 scenes and 1.4 million frames of images, which can meet the richness of training data.

[0110] In some embodiments, the switching mechanism between the short-term prediction model and the long-term prediction model includes: for high-priority motion regions, the detection results are used directly without prediction; for low-priority motion regions, the target's acceleration is calculated, and when the target's acceleration is greater than a preset acceleration threshold, the model switches to a long-term prediction model such as the LSTM model; otherwise, a short-term prediction model such as the Kalman filter prediction model is used. Since the motion region represents the target, the acceleration of the motion region is the same as the acceleration of the target represented by the motion region. The acceleration of the motion region is calculated as follows: Equation (21) in, a t For the acceleration of the moving region, ( v x,t , v y,t () represents the velocity of the moving region at the current moment. v x,t-1 , v y,t-1 ) represents the velocity of the moving region at the previous moment.

[0111] For illustrative purposes, the preset acceleration threshold can be set to 1 pixel / frame by default, or other values.

[0112] In one embodiment of this application, after making a prediction based on the historical perception results of the motion area using a short-term prediction model / long-term prediction model, the method includes: calculating the trajectory deviation value of the motion area based on the prediction result of the motion area and the historical perception result of the motion area; if the trajectory deviation value is greater than a preset deviation threshold, then adjusting the priority of the motion area to a priority higher than the low priority.

[0113] In this embodiment, the trajectory deviation value represents the deviation between the predicted position of the motion region in the prediction result and the historical position of the motion region in the historical perception result, and its calculation method is as follows: Equation (22) in, e t For trajectory deviation value, ( x t , yt () represents the historical location of the movement area. This represents the predicted location of the motion region.

[0114] When the trajectory deviation value exceeds a preset deviation threshold, the priority of the motion area can be temporarily upgraded to high priority, triggering three consecutive full-frame rate detections to correct the prediction deviation. For sudden motion, triggering temporary high-priority processing through the deviation threshold can reduce prediction errors and avoid response delays to sudden targets, so as to respond promptly and effectively to sudden movements of targets in low-priority motion areas, such as a pedestrian standing on the roadside suddenly running across the road, avoiding accidents caused by sudden situations.

[0115] For illustrative purposes, the preset deviation threshold can be set to 5 pixels per frame by default, or other values.

[0116] Please see Figure 4 , Figure 4 This is a block diagram of an image processing apparatus according to an embodiment of this application. The apparatus can be applied to… Figure 1 The implementation environment shown is specifically configured in computer device 120. This device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0117] like Figure 4 As shown, the exemplary image processing device includes: an image acquisition module 410 for acquiring continuous frame images; a motion detection module 420 for performing motion detection on the continuous frame images to obtain each motion region and its velocity in the continuous frame images; a semantic recognition module 430 for determining the target type of each motion region through semantic recognition; an information processing module 440 for determining the priority of each motion region based on its velocity and target type; and a perception recognition module 450 for performing perception recognition on motion regions of different priorities using corresponding perception strategies to obtain perception results.

[0118] The image acquisition module 410, motion detection module 420, semantic recognition module 430, information processing module 440, and perception recognition module 450 can be software modules or hardware modules, and there is no restriction here.

[0119] In some embodiments, the image acquisition module 410 may be an image acquisition device such as a camera or a webcam, or it may be a hardware device for collecting data from the image acquisition device.

[0120] In some embodiments, the motion detection module 420, the semantic recognition module 430, and the perception recognition module 450 may be image processing devices such as ISP (Image Signal Processor), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), and VPU (Vision Processing Unit). The motion detection module 420, the semantic recognition module 430, and the perception recognition module 450 may be independent image processing devices or the same image processing device.

[0121] In other embodiments, the motion detection module 420 may also be a processor or processing chip disposed in an image acquisition device such as a camera or camera.

[0122] In some embodiments, the information processing module 440 may be a computer, computing cluster, microcomputer, embedded computer, processor, or a microprocessor or chip such as MCU (Microcontroller Unit) or ECU (Electronic Control Unit).

[0123] In some embodiments, the image acquisition module 410, motion detection module 420, semantic recognition module 430, information processing module 440, and perception recognition module 450 can be configured in the same electronic device or can be separate electronic devices.

[0124] Please see Figure 5 , Figure 5 This is an architectural diagram of an image processing system provided in an embodiment of this application, such as... Figure 5As shown, the image processing system includes a dynamic region detection module, a priority classification module, a region-based asynchronous scheduling module, a high-priority region processing module, a medium-priority region processing module, a low-priority region processing module, and a temporal fusion and prediction module. The dynamic region detection module performs motion detection on consecutive frames of images to obtain the motion regions and their velocities. The priority classification module determines the target type of each motion region through semantic recognition and determines its priority based on its velocity and target type. The region-based asynchronous scheduling module schedules the high-priority, medium-priority, and low-priority region processing modules to process corresponding motion regions according to their priorities. The high-priority region processing module uses a high-priority corresponding perception strategy to perceive and identify motion regions. The medium-priority region processing module uses a medium-priority corresponding perception strategy to perceive and identify motion regions. The low-priority region processing module uses a low-priority corresponding perception strategy to perceive and identify motion regions. The temporal fusion and prediction module makes predictions based on the historical perception results of low-priority motion regions and compensates for the processing results of the low-priority region processing module based on the prediction results.

[0125] In this embodiment, the dynamic region detection module is the motion detection module 420, the priority classification module can be divided into a semantic recognition module 430 and an information processing module 430, and the high-priority region processing module, the medium-priority region processing module, the low-priority region processing module and the temporal fusion and prediction module can be integrated into the perception and recognition module 450.

[0126] It should be noted that the image processing apparatus and system provided in the above embodiments and the image processing method provided in the above embodiments belong to the same concept. The specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here. In practical applications, the image processing apparatus and system provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus and system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0127] In one embodiment of this application, an electronic device is also provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the image processing methods provided in the above embodiments.

[0128] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer system for an electronic device provided in one embodiment of this application. Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0129] like Figure 6 As shown, the computer system 600 includes a central processing unit 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage section 608 into a random access memory 603, such as performing the methods described in the above embodiments. The random access memory 603 also stores various programs and data required for system operation. The central processing unit 601, the read-only memory 602, and the random access memory 603 are interconnected via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0130] The following components are connected to the input / output interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card and a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0131] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs various functions defined in the system of this application.

[0132] The computer-readable medium shown in the embodiments of this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the image processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0135] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing methods provided in the various embodiments described above.

[0136] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire consecutive frame images; Motion detection is performed on the continuous frame images to obtain each motion region in the continuous frame images and the velocity of each motion region; The target type of each motion region is determined by semantic recognition, and the priority of each motion region is determined according to the speed and target type of each motion region. Different perception strategies are used to perceive and identify motion regions of different priorities, and the perception results are obtained.

2. The image processing method according to claim 1, characterized in that, Motion detection of the consecutive frames of images includes: Perform inter-frame difference operation on the consecutive frame images to determine each first initial motion region in the consecutive frame images; Optical flow calculation is performed on the continuous frame images to determine each second initial motion region and the velocity of each second initial motion region in the continuous frame images; By combining the velocities of each of the first initial motion regions, each of the second initial motion regions, and each of the second initial motion regions, the velocity of each motion region and the velocity of each motion region are obtained.

3. The image processing method according to claim 2, characterized in that, Performing inter-frame difference operations on the consecutive frame images to determine each first initial motion region in the consecutive frame images includes: Calculate the pixel grayscale value and average brightness of each frame for the consecutive frame images; The difference in pixel gray values ​​between adjacent frames is calculated to generate an inter-frame difference image. The inter-frame difference threshold is obtained by weighting the difference based on the average brightness of each frame. The inter-frame difference image is binarized according to the inter-frame difference threshold to obtain a binarized mask, and the consecutive frame images are marked based on the binarized mask to obtain each first initial motion region.

4. The image processing method according to claim 2, characterized in that, Performing optical flow calculations on the consecutive frame images to determine each second initial motion region and the velocity of each second initial motion region in the consecutive frame images includes: Pixels are selected from the consecutive frame images according to a preset window, and an overdetermined system of equations is constructed based on the assumption of constant brightness, with all pixels in the window having the same optical flow as a constraint. The optical flow of each local region is obtained by solving the overdetermined system of equations. A preset speed threshold is compared with the speed of each local region. Based on the comparison result, each local region is filtered to obtain each second initial motion region and the speed of each second initial motion region. The speed of each local region is obtained based on the optical flow of the local region.

5. The image processing method according to claim 1, characterized in that, The target type of each motion region is determined through semantic recognition, and the priority of each motion region is determined according to its speed and target type, including: Semantic recognition is performed on each of the motion regions using a semantic model to obtain the target type of each motion region, wherein the target type has a corresponding confidence level; The priority score of each motion region is obtained by weighting the velocity of each motion region with the confidence level of the corresponding target type. The priority score of each motion region is compared with the preset score range of each priority, and the priority corresponding to each motion region is determined based on the comparison result.

6. The image processing method according to claim 5, characterized in that, The velocity of each motion region and the confidence level of the corresponding target type are weighted and calculated, including: The speed threshold corresponding to each of the motion regions is determined according to the current speed scenario and the target type of each motion region, wherein the current speed scenario includes a high-speed scenario or a low-speed scenario; The velocity of each motion region is normalized based on the velocity threshold corresponding to each motion region to obtain the normalized velocity of each motion region. Based on the normalized speed and preset speed weight of each motion region, the confidence and semantic weight of the target type corresponding to each motion region, the priority score of each motion region is calculated. The semantic weight of the target type is determined based on the false negative risk level of the target type. The higher the false negative risk level of the target type, the greater the semantic weight of the target type.

7. The image processing method according to claim 1, characterized in that, Different perception strategies are used to perceive and identify motion regions of different priorities, including: Under the condition that the priority of the motion region is high, the corresponding computing power is allocated according to the preset weight of the high priority and the area of ​​the motion region, and the motion region is perceived and identified through the full model to obtain the current perception result of the motion region. Under the condition that the priority of the motion region is medium priority, the corresponding computing power is allocated according to the preset weight of the medium priority and the area of ​​the motion region, and the motion region is perceived and identified through a lightweight model to obtain the current perception result of the motion region. When the priority of the motion region is low, the corresponding computing power is allocated according to the preset weight of the low priority and the area of ​​the motion region, and the current perception result of the motion region is predicted based on the historical perception result of the motion region.

8. The image processing method according to claim 7, characterized in that, Predicting the current perception result of the motion region based on the historical perception results of the motion region includes: The acceleration of the motion region is determined based on the historical perception results of the motion region. When the acceleration in the motion region is greater than a preset acceleration threshold, the current perception result of the motion region is obtained by predicting based on the historical perception results of the motion region using a long-term prediction model. When the acceleration of the motion region is less than or equal to a preset acceleration threshold, the current perception result of the motion region is obtained by predicting based on the historical perception results of the motion region using a short-time prediction model.

9. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to acquire consecutive frame images; A motion detection module is used to perform motion detection on the continuous frame images to obtain each motion region in the continuous frame images and the velocity of each motion region. The semantic recognition module is used to determine the target type of each of the motion regions through semantic recognition; The information processing module is used to determine the priority of each motion region based on the speed and target type of each motion region; The perception and recognition module is used to perform perception and recognition on motion areas of different priorities using corresponding perception strategies, and to obtain perception results.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the image processing method as described in any one of claims 1-8.

Images