Frame rate processing method and apparatus for visual sensor, vehicle and program product

Abstract

The application relates to the technical field of image communication, in particular to a frame rate processing method and device of a visual sensor, a vehicle and a program product, wherein the method comprises the following steps: acquiring an original output frame rate of at least one visual sensor, determining a required frame rate of a preset artificial intelligence (AI) model; comparing the original output frame rate with the required frame rate, and determining a current preprocessing algorithm according to a comparison result; and adjusting the original output frame rate based on the current preprocessing algorithm until the original output frame rate is the same as the required frame rate. Therefore, the problems that a visual perception sensor with a fixed frequency cannot meet the requirements of different AI models, the flexibility is low, the adaptive development workload is large and the like are solved, the reusability of data is improved, the efficiency of model training is improved, the stability and reusability of software and models are improved, and the adaptive development workload for new vehicle models (different sensor configurations) is reduced.

Description

Technical Field

[0001] This application relates to the field of image communication technology, and in particular to a frame rate processing method, apparatus, vehicle, and software product for a visual sensor. Background Technology

[0002] Vehicle vision sensors play a crucial role in intelligent connected vehicles. They collect, process, and transmit image data of the vehicle's surrounding environment, providing detailed environmental information to support the decision-making and control of intelligent driving systems.

[0003] In related technologies, visual sensors in automobiles are generally fixed-frequency visual perception sensors. For those with a default frame rate higher than the requirements of AI (Artificial Intelligence) models, the frame rate is reduced by configuring the camera module firmware.

[0004] However, the frame rate of the camera in the relevant technology needs to be configured in advance, and in some cases, it may even be necessary to reselect the camera to meet the requirements of the AI ​​model. Overall, it is not flexible enough. For different sensor configurations, the AI ​​model may need to be retrained, and the adaptation development workload is large, which needs to be solved urgently. Summary of the Invention

[0005] This application provides a frame rate processing method, device, vehicle, and program product for a visual sensor to solve the problems in related technologies, such as the inability of fixed-frequency visual perception sensors to meet the requirements of different AI models, low flexibility, and large workload for adaptation development. It improves the reusability of data during data acquisition, reduces the demand for data acquisition resources, improves the efficiency of model training, improves the stability and reusability of software and models, and reduces the workload of adaptation development for new vehicle models (with different sensor configurations).

[0006] The first aspect of this application provides a frame rate processing method for a visual sensor, comprising the following steps:

[0007] Acquire the raw output frame rate of at least one visual sensor and determine the required frame rate of a preset artificial intelligence (AI) model;

[0008] The original output frame rate and the required frame rate are compared, and the current preprocessing algorithm is determined based on the comparison result.

[0009] The original output frame rate is adjusted based on the current preprocessing algorithm until it is the same as the required frame rate.

[0010] Optionally, in some embodiments, comparing the original output frame rate and the required frame rate, and determining the current preprocessing algorithm based on the comparison result, includes:

[0011] Determine whether the original output frame rate is lower than the required frame rate;

[0012] If the original output frame rate is lower than the required frame rate, the current preprocessing algorithm is a preset frame interpolation algorithm; otherwise, the current preprocessing algorithm is a preset frame extraction algorithm.

[0013] Optionally, in some embodiments, when the original output frame rate is lower than the required frame rate, adjusting the original output frame rate based on the current preprocessing algorithm includes:

[0014] Calculate the first frame rate difference between the required frame rate and the original output frame rate;

[0015] Based on the first frame rate difference, the number of frames of the original output frame rate is increased according to the preset frame interpolation algorithm.

[0016] Optionally, in some embodiments, when the original output frame rate is higher than the required frame rate, adjusting the original output frame rate based on the current preprocessing algorithm includes:

[0017] Calculate the second frame rate difference between the original output frame rate and the required frame rate;

[0018] Based on the second frame rate difference, the number of frames at the original output frame rate is reduced using the preset frame-dropping algorithm.

[0019] Optionally, in some embodiments, when adjusting the original output frame rate based on the current preprocessing algorithm, the method further includes:

[0020] Obtain the timestamp information for each visual sensor;

[0021] The timestamps of multiple visual sensors are aligned according to a preset adjustment algorithm.

[0022] A second aspect of this application provides a frame rate processing apparatus for a visual sensor, comprising:

[0023] The acquisition module is used to acquire the raw output frame rate of at least one visual sensor and determine the required frame rate of a preset artificial intelligence (AI) model.

[0024] The determination module is used to compare the original output frame rate with the required frame rate and determine the current preprocessing algorithm based on the comparison result;

[0025] The processing module is used to adjust the original output frame rate based on the current preprocessing algorithm until the original output frame rate is the same as the required frame rate.

[0026] Optionally, in some embodiments, the determining module is specifically used for:

[0027] Determine whether the original output frame rate is lower than the required frame rate;

[0028] If the original output frame rate is lower than the required frame rate, the current preprocessing algorithm is a preset frame interpolation algorithm; otherwise, the current preprocessing algorithm is a preset frame extraction algorithm.

[0029] Optionally, in some embodiments, when the original output frame rate is lower than the required frame rate, the processing module is specifically used for:

[0030] Calculate the first frame rate difference between the required frame rate and the original output frame rate;

[0031] Based on the first frame rate difference, the number of frames of the original output frame rate is increased according to the preset frame interpolation algorithm.

[0032] Optionally, in some embodiments, when the original output frame rate is higher than the required frame rate, the processing module is specifically used for:

[0033] Calculate the second frame rate difference between the original output frame rate and the required frame rate;

[0034] Based on the second frame rate difference, the number of frames at the original output frame rate is reduced using the preset frame-dropping algorithm.

[0035] Optionally, in some embodiments, when adjusting the original output frame rate based on the current preprocessing algorithm, the processing module is further configured to:

[0036] Obtain the timestamp information for each visual sensor;

[0037] The timestamps of multiple visual sensors are aligned according to a preset adjustment algorithm.

[0038] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the frame rate processing method for a visual sensor as described in the above embodiments.

[0039] A fourth aspect of this application provides a computer program product having a computer program stored thereon that is executed to implement the frame rate processing method for a visual sensor as described in the above embodiments.

[0040] Therefore, this application obtains the raw output frame rate of at least one visual sensor, determines the required frame rate of a preset artificial intelligence (AI) model, compares the raw output frame rate and the required frame rate, determines the current preprocessing algorithm based on the comparison result, and adjusts the raw output frame rate based on the current preprocessing algorithm until the raw output frame rate matches the required frame rate. This solves the problems of fixed-frequency visual perception sensors being unable to meet the requirements of different AI models, having low flexibility, and requiring a large amount of adaptation development work. It improves data reusability during data acquisition, reduces the demand for data acquisition resources, improves model training efficiency, enhances the stability and reusability of software and models, and reduces the adaptation development workload for new vehicle models (with different sensor configurations).

[0041] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0042] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 This is a flowchart of a frame rate processing method for a visual sensor provided according to an embodiment of this application;

[0044] Figure 2 This is a schematic diagram illustrating the processing frame rate according to the first specific embodiment of this application;

[0045] Figure 3 This is a schematic diagram illustrating the processing frame rate according to a second specific embodiment of this application;

[0046] Figure 4 This is a schematic diagram illustrating the processing frame rate according to a third specific embodiment of this application;

[0047] Figure 5 This is a block diagram of a frame rate processing apparatus for a visual sensor according to an embodiment of this application;

[0048] Figure 6 This is a block diagram of a vehicle provided according to an embodiment of this application. Detailed Implementation

[0049] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0050] The following description, with reference to the accompanying drawings, outlines a frame rate processing method, apparatus, vehicle, and software product for a visual sensor according to embodiments of this application. Addressing the issues mentioned in the background art, such as the inability of fixed-frequency visual perception sensors to meet the requirements of different AI models, low flexibility, and significant workload in adaptation and development, this application provides a frame rate processing method for a visual sensor. In this method, the raw output frame rate of at least one visual sensor is acquired, and the required frame rate of a preset artificial intelligence (AI) model is determined. The raw output frame rate and the required frame rate are compared, and a current preprocessing algorithm is determined based on the comparison result. The raw output frame rate is adjusted based on the current preprocessing algorithm until the raw output frame rate matches the required frame rate. This solves the problems of fixed-frequency visual perception sensors failing to meet the requirements of different AI models, low flexibility, and significant workload in adaptation and development. It also improves data reusability during data acquisition, reduces the demand for data acquisition resources, improves model training efficiency, enhances the stability and reusability of software and models, and reduces the workload of adaptation and development for new vehicle models (with different sensor configurations).

[0051] Specifically, Figure 1 This is a schematic flowchart of the frame rate processing method for a visual sensor provided in an embodiment of this application.

[0052] like Figure 1 As shown, the frame rate processing method for this visual sensor includes the following steps:

[0053] In step S101, the raw output frame rate of at least one visual sensor is obtained, and the required frame rate of the preset AI model is determined.

[0054] In this application embodiment, the visual sensor refers to a camera or camera module, and may also be a front-view camera, a surround-view camera, or a rear-view camera.

[0055] The preset AI model in this application works in conjunction with the visual sensor. The preset AI model includes the desired processing frame rate, which is the required frame rate in this application. The required frame rate can be determined comprehensively based on factors such as the model's computing power.

[0056] Understandably, in practical applications, the frame rate required by the AI ​​model should match the output frame rate of the visual sensor to ensure that the AI ​​model can process each frame of image captured by the sensor in a timely manner. Therefore, this embodiment of the application obtains the raw output frame rate of the visual sensor and uses the preprocessing algorithm of this embodiment to automatically adjust the raw output frame rate to match the frame rate required by the AI ​​model.

[0057] Optionally, embodiments of this application may use software tools related to the vision sensor or third-party testing tools to monitor the operating status data of the vision sensor, including the output frame rate.

[0058] Optionally, the determination of the required frame rate of the AI ​​model in this application embodiment can consider multiple factors, including: specific application scenarios, such as video surveillance, autonomous driving, etc.; performance requirement assessment, such as processing speed, accuracy, resource consumption, etc.; hardware limitations, i.e., the limitations of the hardware devices used by the AI ​​model, such as the limited processing power of some hardware devices, which cannot support excessively high frame rates. Therefore, this application embodiment needs to comprehensively consider the above-mentioned multiple factors to accurately determine the required frame rate of the AI ​​model.

[0059] It should be noted that the camera pixels in this embodiment remain fixed. For example, it does not support converting 8MP pixel 20fps data into 2MP pixel 30fps data, but it does support converting 8MP pixel 20fps data into 8MP pixel 30fps data.

[0060] In step S102, the original output frame rate and the required frame rate are compared, and the current preprocessing algorithm is determined based on the comparison result.

[0061] Specifically, this application proposes a preprocessing algorithm to process the raw output frame rate of the visual sensor so that the raw output frame rate of the visual sensor matches the required frame rate of the AI ​​model. As a result, this application does not require adaptation or targeted training. During system operation, it can adaptively adjust and align multiple video input frame rates, and the intelligent driving software itself does not need to perform any adaptation in advance.

[0062] It is understood that, in this embodiment of the application, the original output frame rate of the visual sensor obtained in step S101 and the required frame rate of the AI ​​model are compared, and the current preprocessing algorithm is determined based on the comparison result.

[0063] It should be noted that the preprocessing algorithm in this application embodiment includes: a preset frame interpolation algorithm and a preset frame extraction algorithm. The current preprocessing algorithm is determined based on the comparison result between the original output frame rate and the required frame rate in this application embodiment.

[0064] Optionally, in some embodiments, the original output frame rate and the required frame rate are compared, and the current preprocessing algorithm is determined based on the comparison result, including: determining whether the original output frame rate is lower than the required frame rate; if the original output frame rate is lower than the required frame rate, the current preprocessing algorithm is a preset frame interpolation algorithm; otherwise, the current preprocessing algorithm is a preset frame extraction algorithm.

[0065] In this application embodiment, the preset frame interpolation algorithm is used when the original output frame rate is lower than the required frame rate. Since the original output frame rate is lower than the required frame rate, it is necessary to use the preset frame interpolation algorithm to fill in the original output frame rate of the visual sensor.

[0066] The preset frame-dropping algorithm in this application embodiment is used when the original output frame rate is higher than the required frame rate. Since the original output frame rate is higher than the required frame rate, it is necessary to use the preset frame-dropping algorithm to crop the original output frame rate of the visual sensor.

[0067] Therefore, the embodiments of this application automatically adjust the frame rate of the AI ​​model video input by preprocessing the original output frame rate of the visual sensor, without the need for adaptation or targeted training, thereby improving data reusability and reducing the requirements for data acquisition resources.

[0068] In step S103, the original output frame rate is adjusted based on the current preprocessing algorithm until the original output frame rate is the same as the required frame rate.

[0069] Understandably, after determining the current preprocessing algorithm, the original output frame rate of the visual sensor can be adjusted by matching the obtained preprocessing algorithm so that the original output frame rate of the visual sensor is the same as the required frame rate of the AI ​​model. When the frame rates of the two are the same, the original output frame rate of the visual sensor is then transmitted to the AI ​​model for training.

[0070] For example, Figure 2 This is a schematic diagram illustrating the frame rate processing of the first specific embodiment of this application, as shown below. Figure 2 As shown, in this embodiment of the application, a high-resolution camera is used to collect data at a low frame rate (e.g., 20 frames). The background uses interpolation to increase the frame rate to the mass production solution's frame rate (e.g., 30 frames) before training the model, thereby reducing the consumption of in-vehicle storage resources or reducing the data backhaul bandwidth requirements.

[0071] Optionally, in some embodiments, when the original output frame rate is lower than the required frame rate, the original output frame rate is adjusted based on the current preprocessing algorithm, including: calculating a first frame rate difference between the required frame rate and the original output frame rate; and increasing the number of frames of the original output frame rate based on a preset frame interpolation algorithm according to the first frame rate difference.

[0072] Specifically, if the original output frame rate is lower than the required frame rate, it is necessary to calculate the frame rate difference between the required frame rate and the original output frame rate, and increase the number of frames of the original output frame rate according to the frame rate difference to make up for the gap with the required frame rate.

[0073] Optionally, there are many preset frame interpolation algorithms in the embodiments of this application, such as interpolation. This application does not specifically limit the selection of this algorithm. Those skilled in the art can choose interpolation or other algorithms that have low computational requirements.

[0074] For example, the frame rate of the vehicle's vision sensor is 20 frames, while the AI ​​model requires a frame rate of 30 frames. The domain controller uses interpolation to add 10 frames in real time, which are then input into the AI ​​model and backend algorithm for processing.

[0075] Optionally, in some embodiments, when the original output frame rate is higher than the required frame rate, the original output frame rate is adjusted based on the current preprocessing algorithm, including: calculating a second frame rate difference between the original output frame rate and the required frame rate; and reducing the number of frames of the original output frame rate based on a preset frame-dropping algorithm according to the second frame rate difference.

[0076] Specifically, if the original output frame rate is higher than the required frame rate, it is necessary to calculate the frame rate difference between the original output frame rate and the required frame rate, and then cut off the extra frames from the original output frame rate based on this frame rate difference.

[0077] For example, Figure 3 This is a schematic diagram illustrating the processing frame rate of the second specific embodiment of this application, as shown below. Figure 3 As shown, the frame rate of the vehicle's vision sensor is 35 frames per second, while the AI ​​model requires a frame rate of 30 frames per second. Since the frame rate of the vehicle's vision sensor is higher than the required frame rate, 10 frames are clipped in real time on the domain controller using a frame extraction algorithm before being input into the AI ​​model and the backend algorithm for processing.

[0078] It should be noted that in the above embodiment, the 10 frames that are cut off need to be evenly distributed. The single frame that is removed will cause a gap of 20-30ms in the image, and the AI ​​model needs to be able to handle the image input that is not evenly distributed in time.

[0079] Optionally, in some embodiments, when adjusting the original output frame rate based on the current preprocessing algorithm, the method further includes: obtaining the timestamp information of each visual sensor; and aligning the timestamps of multiple visual sensors according to a preset adjustment algorithm.

[0080] It is understandable that in some cases, AI models need to receive data from multiple different visual sensors with different processing speeds. Considering that network latency and other factors may cause timestamp discrepancies, the embodiments of this application need to align these timestamps to ensure that they represent data at the same or similar points in time.

[0081] Furthermore, each vision sensor typically attaches a timestamp when capturing an image or video frame, which indicates the exact time the frame was captured. This application embodiment needs to obtain these timestamp information from each sensor for subsequent processing.

[0082] Specifically, embodiments of this application can incorporate timestamp processing into the preprocessing algorithm, and synchronize the timestamps of each visual sensor to the same time base by retaining the timestamp information of each frame.

[0083] The preset adjustment algorithm in this application embodiment can be an interpolation or extrapolation method. If the frame rate of a certain sensor is lower than that of other sensors and needs to be aligned with them, an interpolation method can be used to estimate the timestamp of the missing frame, or an extrapolation method can be used to predict the timestamp of future frames. This application does not specifically limit the preset adjustment algorithm, and those skilled in the art can set it based on the actual situation.

[0084] For example, Figure 4 This is a schematic diagram illustrating the processing frame rate of the third specific embodiment of this application, as shown below. Figure 4 As shown, the output frame rates of different vision sensors in the vehicle are different (e.g., 35 frames for the front-view camera, 25 frames for the surround-view camera, and 20 frames for the rear-view camera). The frame rate is adjusted to the required frame rate of the model in real time on the domain controller through interpolation or frame extraction algorithm, and the timestamps between different sensors are aligned before being input to the AI ​​model and back-end algorithm for processing.

[0085] Therefore, this application automatically completes or crops image frames from the visual sensor through simple interpolation or frame extraction algorithms, adjusting videos with different frame rates to the frame rate required by the AI ​​model for processing (running scenario) or training (data acquisition scenario). During system operation, it can adaptively adjust and align the frame rates of multiple video inputs. The intelligent driving software itself does not need to be adapted in advance, which improves the stability and reusability of the software and model, and reduces the workload of adaptation development for new vehicle models (different sensor configurations).

[0086] Furthermore, the frame rate processing method for the visual sensor in this application embodiment can be extended to other sensor data processing algorithms. For example, CAN (Controller Area Network) and vehicle-mounted ETH (Ethernet) messages can use this method to reduce the transmission frequency, thereby reducing network load and CPU (Central Processing Unit) load at the transmitting end.

[0087] The frame rate processing method for visual sensors proposed in this application involves acquiring the raw output frame rate of at least one visual sensor, determining the required frame rate of a preset artificial intelligence (AI) model, comparing the raw output frame rate with the required frame rate, determining the current preprocessing algorithm based on the comparison result, and adjusting the raw output frame rate based on the current preprocessing algorithm until the raw output frame rate matches the required frame rate. This solves the problems of fixed-frequency visual perception sensors failing to meet the requirements of different AI models, exhibiting low flexibility, and requiring significant adaptation development work. It improves data reusability during data acquisition, reduces the demand for data acquisition resources, increases model training efficiency, enhances the stability and reusability of software and models, and reduces the adaptation development workload for new vehicle models (with different sensor configurations).

[0088] Next, with reference to the accompanying drawings, a frame rate processing apparatus for a visual sensor according to an embodiment of this application is described.

[0089] Figure 2 This is a block diagram of a frame rate processing device for a visual sensor according to an embodiment of this application.

[0090] like Figure 2 As shown, the frame rate processing device 10 of the visual sensor includes: an acquisition module 100, a determination module 200, and a processing module 300.

[0091] Specifically, the acquisition module 100 is used to acquire the raw output frame rate of at least one visual sensor and determine the required frame rate of the preset artificial intelligence AI model; the determination module 200 is used to compare the raw output frame rate and the required frame rate and determine the current preprocessing algorithm based on the comparison result; the processing module 300 is used to adjust the raw output frame rate based on the current preprocessing algorithm until the raw output frame rate is the same as the required frame rate.

[0092] Optionally, in some embodiments, the determining module 200 is specifically used to: determine whether the original output frame rate is lower than the required frame rate; if the original output frame rate is lower than the required frame rate, the current preprocessing algorithm is a preset frame interpolation algorithm; otherwise, the current preprocessing algorithm is a preset frame extraction algorithm.

[0093] Optionally, in some embodiments, when the original output frame rate is lower than the required frame rate, the processing module 300 is specifically used to: calculate a first frame rate difference between the required frame rate and the original output frame rate; and increase the number of frames of the original output frame rate based on a preset frame interpolation algorithm according to the first frame rate difference.

[0094] Optionally, in some embodiments, when the original output frame rate is higher than the required frame rate, the processing module 300 is specifically used to: calculate a second frame rate difference between the original output frame rate and the required frame rate; and reduce the number of frames of the original output frame rate based on a preset frame-dropping algorithm according to the second frame rate difference.

[0095] Optionally, in some embodiments, when adjusting the original output frame rate based on the current preprocessing algorithm, the processing module 300 is further configured to: obtain the timestamp information of each visual sensor; and align the timestamps of multiple visual sensors according to a preset adjustment algorithm.

[0096] It should be noted that the foregoing explanation of the frame rate processing method embodiment for the visual sensor also applies to the frame rate processing device for the visual sensor in this embodiment, and will not be repeated here.

[0097] The frame rate processing device for a visual sensor proposed in this application acquires the original output frame rate of at least one visual sensor, determines the required frame rate of a preset artificial intelligence (AI) model, compares the original output frame rate and the required frame rate, determines the current preprocessing algorithm based on the comparison result, and adjusts the original output frame rate based on the current preprocessing algorithm until the original output frame rate matches the required frame rate. This solves the problems of fixed-frequency visual perception sensors failing to meet the requirements of different AI models, having low flexibility, and requiring significant adaptation development work. It improves data reusability during data acquisition, reduces the demand for data acquisition resources, increases model training efficiency, enhances the stability and reusability of software and models, and reduces the adaptation development workload for new vehicle models (with different sensor configurations).

[0098] Figure 6 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0099] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0100] When the processor 602 executes the program, it implements the frame rate processing method for the visual sensor provided in the above embodiments.

[0101] Furthermore, the vehicle also includes:

[0102] Communication interface 603 is used for communication between memory 601 and processor 602.

[0103] The memory 601 is used to store computer programs that can run on the processor 602.

[0104] The memory 601 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0105] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0107] The processor 602 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0108] This application also provides a computer program product on which a computer program is stored, which, when executed, implements the above-described frame rate processing method for a visual sensor.

[0109] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0110] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0111] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0112] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0113] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0114] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A frame rate processing method of a vision sensor, characterized by, The method comprises the following steps: acquiring a raw output frame rate of at least one visual sensor and determining a required frame rate of a preset artificial intelligence (AI) model; comparing the raw output frame rate and the required frame rate and determining a current preprocessing algorithm according to a comparison result; adjusting the raw output frame rate based on the current preprocessing algorithm until the raw output frame rate is the same as the required frame rate; the comparing the raw output frame rate and the required frame rate and determining the current preprocessing algorithm according to the comparison result comprises: judging whether the raw output frame rate is lower than the required frame rate; if the raw output frame rate is lower than the required frame rate, the current preprocessing algorithm is a preset frame filling algorithm, otherwise, the current preprocessing algorithm is a preset frame skipping algorithm; when the raw output frame rate is lower than the required frame rate, the adjusting the raw output frame rate based on the current preprocessing algorithm comprises: calculating a first frame rate difference between the required frame rate and the raw output frame rate; and increasing the number of frames of the raw output frame rate based on the preset frame filling algorithm according to the first frame rate difference.

2. The method of claim 1, wherein, when the raw output frame rate is higher than the required frame rate, the adjusting the raw output frame rate based on the current preprocessing algorithm comprises: calculating a second frame rate difference between the raw output frame rate and the required frame rate; decreasing the number of frames of the raw output frame rate based on the preset frame skipping algorithm according to the second frame rate difference.

3. The method of claim 1, wherein, when the raw output frame rate is adjusted based on the current preprocessing algorithm, the method further comprises: acquiring timestamp information of each visual sensor; aligning timestamps of the multiple visual sensors according to a preset adjustment algorithm.

4. A frame rate processing apparatus of a vision sensor, characterized by, The method comprises: an acquiring module configured to acquire a raw output frame rate of at least one visual sensor and determine a required frame rate of a preset artificial intelligence (AI) model; a determining module configured to compare the raw output frame rate and the required frame rate and determine a current preprocessing algorithm according to a comparison result; a processing module configured to adjust the raw output frame rate based on the current preprocessing algorithm until the raw output frame rate is the same as the required frame rate; the determining module is specifically configured to: judge whether the raw output frame rate is lower than the required frame rate; in a case where the raw output frame rate is lower than the required frame rate, the current preprocessing algorithm is a preset frame filling algorithm, otherwise, the current preprocessing algorithm is a preset frame skipping algorithm; when the raw output frame rate is lower than the required frame rate, the processing module is specifically configured to: calculate a first frame rate difference between the required frame rate and the raw output frame rate; and increase the number of frames of the raw output frame rate based on the preset frame filling algorithm according to the first frame rate difference.

5. A vehicle characterized by comprising: The method comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, the processor executes the program to implement the frame rate processing method of the visual sensor according to any one of claims 1-3.

6. A computer program product comprising a computer program, characterized in that, The computer program is executed to implement the frame rate processing method of the visual sensor according to any one of claims 1-3.

Images