Method and device for processing image data of image sensor

Abstract

The present application relates to a method and device for processing image data of an image sensor. The method includes: receiving first image data from an image sensor, the first image data being one of multiple image data that can be generated by the image sensor scanning a physical area corresponding to an acquisition area within a scanning cycle, and the acquisition area represents the acquisition range of the image sensor; performing image processing on the first image data to obtain second image data; and outputting the second image data. The method and device for processing image data of an image sensor in the present application can advance the processing flow of image data generated by the image sensor, thereby reducing the end-to-end delay from the image sensor generating the image data to the vehicle actuator performing the operation. The method and device for processing image data of an image sensor in the present application are particularly suitable for advanced driver assistance systems (ADAS) and robotic systems of intelligent connected vehicles.

Description

Technical Field

[0001] The present application relates to the field of intelligent connected vehicles, and in particular to a method and device for processing image data of an image sensor. Background Art

[0002] Perception recognition plays an important role in the Advanced Driving Assistance System (ADAS) and Autonomous Driving System of intelligent connected vehicles. In order to realize the perception recognition function, intelligent connected vehicles are equipped with a variety of image sensors. Common examples of on-board image sensors for intelligent perception recognition are cameras, LiDAR (Light Detection and Ranging), millimeter wave radar, etc. The "image data" they obtain is rich in information, so intelligent connected vehicles can realize recognition functions through these image data.

[0003] In intelligent connected vehicles, image data arrives at the advanced driver assistance system or autonomous driving system from the image sensor, and after image processing, algorithm recognition, driving decision-making and other steps, it is finally transformed into operation control instructions for each actuator of the vehicle to achieve driving control. In this process, for safety reasons, reducing the end-to-end delay from the image sensor generating image data to the vehicle actuator executing the operation is a goal that advanced driver assistance systems and autonomous driving systems continue to pursue.

[0004] Among the on-board sensor image sensors, there are many image sensors with "scanning" sensing methods, such as lidar and cameras. This scanning image sensor does not generate all the image data of a frame of image in the acquisition area at the same time, but generates all the image data of a frame of image in sequence within a period of time, for example, by scanning by row or column. At present, the on-board image signal processor (ISP) and subsequent algorithm platform process the frame of image after waiting to receive all the image data of a frame of image. This processing method has a great impact on end-to-end latency. Summary of the invention

[0005] In view of this, the present application provides a method and apparatus for processing image data of an image sensor, which can reduce the end-to-end delay from the image sensor generating the image data to the vehicle actuator executing the operation.

[0006] In a first aspect, a method for processing image data of an image sensor is provided, the method comprising the following steps: receiving first image data from an image sensor, the first image data being one of a plurality of image data that can be generated by the image sensor by scanning a physical area corresponding to an acquisition area within one scanning cycle, the acquisition area representing an acquisition range of the image sensor; performing image processing on the first image data to obtain second image data; and outputting the second image data.

[0007] By receiving one of the multiple image data that can be generated by the image sensor in a scanning cycle when scanning the physical area corresponding to the acquisition area, and performing image processing on the received first image data, the image processing flow of the first image data generated by the image sensor can be advanced, thereby reducing the end-to-end delay from the image sensor generating the image data to the vehicle actuator performing the operation. In other words, when a portion of the first image data of a frame of image that can be generated by the scanning acquisition area is received, image processing can be performed on the portion of the first image data, without having to wait until all the first image data of the entire frame of the image are received before starting the image processing flow of the image data.

[0008] In a possible implementation manner, the image sensor is a camera, and the image is a two-dimensional plane image.

[0009] The acquisition area of ​​a camera is usually referred to as a target surface. The frame rate of a camera is generally 30 Hz (Hertz). In other words, the time it takes for the camera to expose the first line of the target surface to the last line of the target surface to form a frame of image is 33 ms (milisecond). It is usually necessary to wait for all the first image data of the entire frame of image to be generated and transmitted before processing the image data of the received entire frame of image. That is, after the camera starts scanning the target surface, it is necessary to wait at least 33 ms to process the first image data generated thereby. However, through the method of processing image data of the present application, for example, after receiving the first first image data of a frame of image, the first image data can be processed immediately without having to wait until all the first image data of the entire frame of image are received. This can advance the image processing flow of the first image data of the plane image, save the processing time of the first image data of each frame of the plane image, and thus reduce the end-to-end delay from the generation of the first image data by the camera to the execution of the operation by the vehicle actuator.

[0010] In a possible implementation, the image sensor is a laser radar, and the image is a three-dimensional point cloud. The acquisition area of ​​the laser radar is usually called a scanning area.

[0011] The frame rate of the laser radar is generally 10Hz or 20Hz. That is to say, the time it takes for the laser radar to scan from the first column of the scanning area to the last column of the scanning area to form the first image data of a frame of point cloud is usually 100ms or 50ms. Similar to the processing of image data of a plane image, it is usually necessary to wait for the first image data of the entire frame of point cloud to be generated and transmitted before processing the first image data of the received entire frame of point cloud. However, through the method of processing image data of the present application, for example, after receiving the first first image data of a frame of point cloud, the first image data can be processed immediately without having to wait until all the first image data of the entire frame of point cloud are received. This can advance the image processing flow of the first image data of the point cloud, save the processing time of the first image data of each frame of point cloud, and thus reduce the end-to-end delay from the generation of the first image data by the laser radar to the execution of the operation by the vehicle actuator.

[0012] In combination with the first aspect, in a possible implementation method, the acquisition area includes multiple sub-acquisition areas; the image processing of the first image data includes: after receiving the first image data contained in the first data group A, image processing is performed on all the first image data contained in the first image data group A as units, and the first image data group A is a collection of the first image data generated by the image sensor scanning a physical area corresponding to the sub-acquisition area.

[0013] By defining each first image data group by each sub-acquisition area of ​​the acquisition area, multiple first image data of a frame of image are sorted in advance. After receiving the first image data contained in the first data group A, image processing is performed on all the first image data contained in the first image data group A, that is, by performing image processing on the first image data group as a unit, the image processing flow of the first image data can be advanced while avoiding excessive increase in the complexity of subsequent processing and the increase in subsequent processing time caused by the increase. In addition, by defining each first image data group by a sub-acquisition area, it can be applicable to image sensors with different resolutions.

[0014] In combination with the first aspect, in a possible implementation manner, the sizes and numbers of the multiple sub-collection areas are preset.

[0015] In a possible implementation manner, the number of sub-collection areas is selected from 2 to 4.

[0016] In a possible implementation, the sizes of the sub-collection areas are equal.

[0017] In combination with the first aspect, in a possible implementation manner, the method further includes the following steps: receiving a division strategy, and presetting the sizes and numbers of the multiple sub-collection areas according to the division strategy.

[0018] By receiving a division strategy to pre-set the size and number of multiple sub-capture areas, the image processing process of multiple first image data can be flexibly adjusted according to the actual needs of the application scenario, so that the method of processing image sensor image data of the present application is adapted to various application scenarios.

[0019] With reference to the first aspect, in a possible implementation manner, the sub-collection area is a rectangle, and a size of the sub-collection area is defined by coordinates of four corners of the rectangle.

[0020] By defining the size of the sub-acquisition area by the coordinates of the four corners of the sub-acquisition area, each sub-acquisition area can be flexibly divided in a particularly simple and intuitive manner.

[0021] In a second aspect, a method for processing image data of an image sensor is provided, the method comprising the following steps: receiving second image data in sequence, the second image data being obtained by performing image processing on first image data, the first image data being one of a plurality of image data that can be generated by the image sensor scanning a physical area corresponding to an acquisition area within one scanning cycle, the acquisition area representing an acquisition range of the image sensor; extracting feature data from the second image data in sequence; performing fusion and recognition processing on each feature data; and outputting an image recognition result.

[0022] By extracting feature data from the received second image data in sequence while receiving the second image data, the image recognition process of the second image data can be advanced, thereby reducing the end-to-end delay from the image sensor generating the first image data to the vehicle actuator performing the operation. In other words, when a portion of the second image data of a frame of image is received, feature data can be extracted as part of the recognition algorithm processing process for the portion of the second image data of the frame of image, without having to wait until all the second image data of the entire frame of image is received before starting feature extraction.

[0023] In addition, by dividing the traditional recognition algorithm processing process into a feature extraction process and a fusion recognition process, for example, in the feature extraction process, feature data of the second image data originating from different parts of the acquisition area are extracted separately, and in the fusion recognition, the feature data extracted from all the second image data are fused and recognized, the stability and accuracy of the image recognition results of the fusion recognition can be guaranteed.

[0024] In conjunction with the second aspect, in a possible implementation manner, the image sensor is a camera, and the image is a two-dimensional plane image. In conjunction with the second aspect, in a possible implementation manner, the image sensor is a laser radar, and the image is a three-dimensional point cloud.

[0025] In combination with the second aspect, in a possible implementation manner, the second image data are grouped; and the respective feature data are extracted from each group of the second image data.

[0026] By grouping the second image data, a plurality of second image data of a frame of image are sorted in advance. By extracting the respective feature data from each group of the second image data, that is, extracting feature data with a group of second image data as a unit, the image recognition process of the image data can be advanced while avoiding excessive increase in the complexity of subsequent processing and the increase in subsequent processing time caused by the increase.

[0027] In combination with the second aspect, in a possible implementation manner, the number of second image data in each group of second image data is preset.

[0028] In combination with the second aspect, in a possible implementation, the acquisition area includes multiple sub-acquisition areas, and the second image data are grouped according to the multiple sub-acquisition areas. A group of second image data is a group of first image data that has been image processed, and the group of first images is a collection of the first image data generated by the image sensor scanning a physical area corresponding to the sub-acquisition area.

[0029] In combination with the second aspect, in a possible implementation manner, the method further includes the following steps: receiving a division strategy, and presetting the sizes and numbers of the multiple sub-collection areas according to the received division strategy.

[0030] By receiving a division strategy to pre-set the size and number of multiple sub-acquisition areas, thereby adjusting the grouping of the image data, the process of extracting feature data from multiple image data can be flexibly adjusted according to the actual needs of the application scenario, so that the method of processing image sensor image data of the present application is adapted to various application scenarios.

[0031] In combination with the second aspect, in a possible implementation manner, the sub-collection area is a rectangle, and the size of the sub-collection area is defined by coordinates of four corners of the rectangle.

[0032] By defining the size of the sub-acquisition region by the coordinates of the four corners of the sub-acquisition region, each sub-acquisition region can be flexibly divided in a particularly simple and intuitive manner.

[0033] In combination with the second aspect, in a possible implementation, the extracted feature data includes convolution processing and pooling (also known as aggregation) processing.

[0034] In one possible implementation, the convolution process and the pooling process are performed alternately more than once. In one possible implementation, the convolution process includes one or more sub-convolution processes. In one possible implementation, the number of sub-convolution processes in the convolution process is a natural number selected from 1 to 3.

[0035] In combination with the second aspect, in a possible implementation manner, the fusion recognition includes feature fusion processing.

[0036] Through the feature fusion processing, each feature data extracted from the second image data from each acquisition area can be effectively fused into the form of feature data of the entire frame image.

[0037] In combination with the second aspect, in a possible implementation manner, the fusion identification also includes full connection processing.

[0038] Through full connection processing, it is possible to perform global analysis and recognition on the features of the image data of the entire frame of the image.

[0039] In a possible implementation, the feature fusion process includes a series of feature fusion (concat) processes. In a possible implementation, the full connection process includes one or more sub-full connection processes.

[0040] In a third aspect, an image data processing device is provided, comprising: a receiving module, used to receive first image data from an image sensor, the first image data being one of multiple image data that can be generated by the image sensor by scanning a physical area corresponding to an acquisition area within one scanning cycle, and the acquisition area represents an acquisition range of the image sensor; and an image processing module, used to perform image processing on the first image data to obtain second image data, and to output the second image data.

[0041] In combination with the third aspect, in a possible implementation method, the acquisition area includes multiple sub-acquisition areas; the image processing of the first image data includes: the image processing module is also used to perform image processing on all the first image data contained in the first image data group A as units after the receiving module receives the first image data contained in the first data group A, and the first image data group A is a collection of the first image data generated by the image sensor scanning a physical area corresponding to the sub-acquisition area.

[0042] In combination with the third aspect, in a possible implementation manner, the sizes and numbers of the multiple sub-collection areas are preset.

[0043] In combination with the third aspect, in a possible implementation manner, the sub-collection area is a rectangle, and the size of the sub-collection area is defined by coordinates of four corners of the rectangle.

[0044] In combination with the third aspect, in a possible implementation manner, the receiving module is further used to receive a division strategy, and the image data processing module is further used to pre-set the sizes and quantities of the multiple sub-acquisition areas according to the division strategy.

[0045] Since the device of the third aspect of the present application can execute the method of the first aspect, the advantages and benefits of the device of the third aspect are similar to those of the first aspect. Please refer to the relevant description of the first aspect and will not repeat them here.

[0046] In a fourth aspect, an image recognition device is provided, comprising: a receiving module, used to sequentially receive second image data, where the second image data is obtained by performing image processing on the first image data, where the first image data is one of multiple image data that can be generated by the image sensor scanning a physical area corresponding to an acquisition area within one scanning cycle, and the acquisition area represents an acquisition range of the image sensor; a feature extraction module, used to sequentially extract feature data from the second image data; and a fusion recognition module, used to perform fusion recognition processing on each feature data, and to output an image recognition result.

[0047] In combination with the fourth aspect, in a possible implementation manner, the number of second image data in each group of second image data is preset.

[0048] In combination with the fourth aspect, in a possible implementation, the acquisition area includes multiple sub-acquisition areas, and the second image data are grouped according to the multiple sub-acquisition areas. A group of second image data is a group of first image data that has been image processed, and the group of first images is a collection of the first image data generated by the image sensor scanning a physical area corresponding to the sub-acquisition area.

[0049] In combination with the fourth aspect, in a possible implementation manner, the sizes and numbers of the multiple sub-collection areas are preset.

[0050] In combination with the fourth aspect, in a possible implementation manner, the receiving module is further used to receive a division strategy, and the feature extraction module is further used to pre-set the sizes and quantities of the multiple sub-acquisition areas according to the division strategy.

[0051] In conjunction with the fourth aspect, in a possible implementation manner, the sub-collection area is a rectangle, and a size of the second sub-collection area is defined by coordinates of four corners of the rectangle.

[0052] In combination with the fourth aspect, in a possible implementation, the feature extraction module includes a convolution layer and a pooling layer.

[0053] In a possible implementation, more than one convolution layer and pooling layer are alternately configured. In a possible implementation, the convolution layer includes one or more sub-convolution layers. In a possible implementation, the number of sub-convolution layers in the convolution layer is a natural number selected from 1 to 3.

[0054] In combination with the fourth aspect, in a possible implementation manner, the fusion recognition module includes a feature fusion layer.

[0055] In combination with the fourth aspect, in a possible implementation manner, the fusion recognition module also includes a fully connected layer.

[0056] In a possible implementation, the feature fusion processing module includes a series of feature fusion (concat) layers. In a possible implementation, the fully connected layer includes one or more sub-fully connected layers.

[0057] In combination with the fourth aspect, in a possible implementation manner, the receiving module is further used to receive a division strategy, and the feature extraction module is further used to pre-set the sizes and quantities of the multiple second sub-acquisition areas according to the division strategy.

[0058] Since the device of the fourth aspect of the present application can execute the method of the second aspect, the advantages and benefits of the device of the fourth aspect are similar to those of the second aspect. Please refer to the relevant description of the second aspect and will not repeat them here.

[0059] In a fifth aspect, an image sensor image data processing system is provided, comprising any image data processing device as in the third aspect and any image recognition device as in the fourth aspect.

[0060] In combination with the fifth aspect, in a possible implementation, the image sensor image data processing system also includes a partition management module, which is used to provide a partition strategy to the image data processing device and the image recognition device, and the partition strategy is used to pre-set the size and number of multiple sub-acquisition areas of the acquisition area.

[0061] Since the system of the fifth aspect includes the devices of the third and fourth aspects, the advantages and benefits of the system of the fifth aspect will include the advantages and benefits of the third and fourth aspects. Please refer to the relevant descriptions of the third and fourth aspects and will not be repeated here.

[0062] In a sixth aspect, a driving system is provided, comprising any one of the image sensor image data processing systems of the fifth aspect and a driving decision unit; wherein the driving decision unit is connected to the image sensor image data processing system, and is used to execute behavior decisions and motion planning and output operation instructions based on the image recognition results output by the image sensor image data processing system.

[0063] By adopting any one of the image sensor image data processing systems of the fifth aspect, the driving system of the present application can process the image data in advance, save processing time, and thus reduce the end-to-end delay from the image sensor generating image data to the vehicle actuator executing the operation.

[0064] In one possible implementation, the driving system is an advanced driver assistance system. In another possible implementation, the driving system is an autonomous driving system.

[0065] In the seventh aspect, a vehicle is provided, comprising an image sensor, any one of the driving systems in the sixth aspect, an electronic control unit and an actuator connected in sequence; wherein the image sensor is used to sense the vehicle environment in a scanning manner and output first image data; and the electronic control unit is used to control the actuator to perform operations according to the operating instructions of the driving system.

[0066] By adopting any one of the driving systems of the sixth aspect mentioned above, the vehicle of the present application can process the process in advance and save processing time, thereby reducing the end-to-end delay from the image sensor generating image data to the vehicle actuator executing the operation.

[0067] In an eighth aspect, a computing device is provided, comprising: at least one processor; and at least one memory connected to the processing unit and storing program instructions, wherein the program instructions, when executed by the at least one processor, enable the at least one processor to execute the method described in any one of the first and third aspects.

[0068] Since the processor in the computing device of the present application can execute any one of the methods for processing image sensor image data in the first and second aspects above, the advantages and benefits of the computing device are also similar to those of the first and second aspects. Please refer to the relevant descriptions of the first and second aspects and will not be repeated here.

[0069] In a ninth aspect, a computer-readable storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer, the computer executes any one of the methods for processing image sensor image data in the first and second aspects.

[0070] Since the computer-readable storage medium of the present application can enable a computer to execute any one of the methods for processing image sensor image data in the first and second aspects above, the advantages and benefits of the computer-readable storage medium are also similar to the advantages and benefits of the first and second aspects. Please refer to the relevant descriptions of the first and second aspects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The following further illustrates the various features of the present application and the connections between the various features with reference to the accompanying drawings. The accompanying drawings are all exemplary, and some features are not shown in actual proportion. Some drawings may omit features that are conventional in the field involved in the present application and are not necessary for the present application, or additional features that are not necessary for the present application may be shown. The combination of the various features shown in the accompanying drawings is not intended to limit the present application. In addition, throughout this specification, the same figure numerals refer to the same content. The specific description of the drawings is as follows:

[0072] Figure 1 is a schematic diagram of an image data processing solution in the prior art;

[0073] Figure 2 Schematic diagram of an image data processing solution of the prior art and an embodiment of the present application, wherein the solution of the prior art is located at the upper part, and the solution of the embodiment of the present application is located at the lower part;

[0074] Figure 3 is a schematic structural diagram of an image sensor image data processing system according to one embodiment of the present application;

[0075] Figure 4 is a structural schematic diagram of an image data processing device according to an embodiment of the present application;

[0076] Figure 5 is a schematic structural diagram of an image recognition device according to an embodiment of the present application;

[0077] Figure 6 is a flowchart of a method for processing image data according to one embodiment of the present application;

[0078] Figure 7 It is a schematic diagram of dividing the collection area using the division strategy in the implementation mode of the present application;

[0079] Figure 8 yes Figure 7 Schematic diagram of the division of each sub-collection area of ​​the image sensor collection area;

[0080] Fig. 9 is a flowchart of a feature extraction and fusion recognition process in a method for processing image data of an image sensor according to an embodiment of the present application; and

[0081] Fig.10 is a schematic diagram of an image data processing scheme of the prior art and another embodiment of the present application, wherein the scheme of the prior art is located at the upper part, and the scheme of the embodiment of the present application is located at the lower part;

[0082] Fig.11is a schematic diagram of an image data processing scheme of the prior art and another embodiment of the present application, wherein the scheme of the prior art is located at the upper part, and the scheme of the embodiment of the present application is located at the lower part;

[0083] Fig.12 is a flowchart of a method for processing image data of an image sensor according to another embodiment of the present application;

[0084] Fig.13 is a schematic structural diagram of a driving system according to an embodiment of the present application;

[0085] Fig.14 is a schematic structural diagram of a vehicle according to an embodiment of the present application; and

[0086] Fig.15 It is a schematic diagram of the structure of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0087] <Definition>

[0088] In the following description, the numbers representing the steps, such as S101, S102, etc., do not necessarily mean that the steps must be executed in this manner. If permitted, the order of the previous and next steps can be interchanged, or they can be executed simultaneously.

[0089] The words "first, second, third" or "region A, region B, region C" and the like in the specification and claims are only used to distinguish similar objects and do not indicate a specific ordering of the objects. It is understandable that the specific order or sequence can be interchanged where permitted so that the implementation method of the present application described herein can be implemented in an order other than that illustrated or described herein. For example, the terms "first image data" and "second image data" used in the present application are both image data, but the "second image data" is obtained by image processing the "first image data", so "first" and "second" are used to distinguish the two.

[0090] The term "comprising" as used in the description and claims should not be interpreted as being limited to what is listed thereafter; it does not exclude other elements or steps. Therefore, it should be interpreted as specifying the presence of the features, integers, steps or components mentioned, but does not exclude the presence or addition of one or more other features, integers, steps or components and groups thereof. Therefore, the expression "a device comprising means A and B" should not be limited to a device consisting of components A and B only.

[0091] References to "one embodiment" or "an embodiment" in the specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various places in this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. In addition, in one or more embodiments, the particular features, structures, or characteristics can be combined in any appropriate manner, as will be apparent to one of ordinary skill in the art from this disclosure.

[0092] The terms "optional", "optionally" and "optional" used in the specification mean that the features they modify can be omitted in some embodiments, but exist in some alternative embodiments.

[0093] The term "imaging sensor" used in the specification and claims includes cameras and laser radars. An image sensor is used to generate multiple image data by scanning a physical area corresponding to a collection area in one scanning cycle. The collection area refers to the collection range of the image sensor. The collection area of ​​a camera is also called a target surface. The collection area of ​​a laser radar is also called a scanning area.

[0094] The term "image processing" used in the specification and claims refers to the technology of analyzing and processing images with a computing device to achieve the desired results. The image processing applicable to image data of different types of images is different. For example, image processing for image data of two-dimensional plane images may include but is not limited to black level compensation, lens shading correction, bad pixel correction, color interpolation, Bayesian domain denoising, automatic white balance, color correction, gamma correction and color space conversion (RGB to YUV). Image processing for image data of three-dimensional point clouds may include but is not limited to filtering, downsampling and outlier removal. Therefore, the "image processing" involved in the method implementation of the present application may include one or more sub-image processing, and the "image processing module" involved in the device implementation of the present application may include one or more sub-image processing modules.

[0095] The term "feature extraction" or "extraction of feature data" as used in the specification and claims refers to the technology of constructing a variety of informative but non-redundant feature data from image data after image processing. The feature extraction applicable to image data of different types of images is different. For example, feature extraction of image data for two-dimensional plane images may include but is not limited to convolution and pooling. Feature extraction of image data for three-dimensional point clouds may include but is not limited to feature vector extraction.

[0096] The term "fusion and recognition" used in the specification and claims refers to a technology that fuses all feature data extracted from image data into a whole, performs recognition analysis based on the whole, and outputs an image recognition result. The fusion and recognition applicable to image data of different types of images is different. For example, the fusion and recognition of image data for two-dimensional plane images may include but is not limited to feature fusion, fully connected, and output. The fusion and recognition of image data for three-dimensional point clouds may include but is not limited to feature point matching.

[0097] The term "feature fusion" used in the specification and claims is the process of fusing all feature data extracted from the image data of the entire acquisition area into one. The feature fusion involved in this application is an early fusion performed before obtaining the image recognition result, which may include but is not limited to serial feature fusion (concat) and parallel fusion (add).

[0098] <Technical Background>

[0099] At present, many on-vehicle image sensors have a "scanning" sensing method. Such a scanning image sensor does not generate all the first image data of a frame of image in the acquisition area at the same time, but instead generates all the first image data of a frame of image sequentially within a period of time by scanning the acquisition area in rows or columns. Figure 1As shown, in the existing method for processing image data of an image sensor, starting from time t0, the image sensor generates and transmits the first first image data of a frame of image, and only when the last first image data of the frame of image is transmitted at time t1, image processing is performed on all the first image data of the entire frame of image. Then, at time t2, when the image processing of the first image data of the entire frame of image is completed, the recognition algorithm processing is started on the second image data (i.e., the first image data after image processing), until the recognition algorithm processing of the entire frame of image is completed at time t3.

[0100] The inventors of the present application have found that such a method of processing image sensor image data has the following defects. The image processing process must wait until the image sensor scans the entire acquisition area to generate all the first image data and transmits them. The image recognition process must also wait until all the first image data are processed and converted into second image data. These waiting times each delay the processing flow of the image sensor image data, resulting in a large end-to-end delay from the image sensor generating the first image data to the vehicle actuator performing the operation. In other words, since the image processing and recognition algorithm processing are performed in units of the first or second image data of the entire frame of the image, it is impossible to process the received first or second image data while receiving the first or second image data of the frame, which reduces the speed of processing the image sensor image data and increases the end-to-end delay.

[0101] The inventors of the present application have found that such a method of processing image sensor image data has the following defects. The image processing process must wait until the image sensor scans the entire acquisition area to generate all the first image data and transmits them. The image recognition process must also wait until all the first image data are processed and converted into second image data. These waiting times each delay the processing flow of the image sensor image data, resulting in a large end-to-end delay from the image sensor generating the first image data to the vehicle actuator performing the operation. In other words, since the image processing and recognition algorithm processing are performed in units of the first or second image data of the entire frame of the image, it is impossible to process the received first or second image data while receiving the first or second image data of the frame, which reduces the speed of processing the image sensor image data and increases the end-to-end delay.

[0102] <Technical Concept>

[0103] In view of this, an embodiment of the present application provides a method for processing image data of an image sensor, comprising: receiving first image data from an image sensor, the first image data being one of multiple image data that can be generated by the image sensor by scanning a physical area corresponding to an acquisition area within one scanning cycle, the acquisition area representing the acquisition range of the image sensor; performing image processing on the first image data to obtain second image data; and outputting the second image data.

[0104] The following combination Figure 2 The concept of the method for processing image sensor image data of the present application is further explained. For the sake of intuition, Figure 2 The existing solution (located at Figure 2 Upper part) and the present application (located at Figure 2 lower part).

[0105] exist Figure 2 In the implementation of the method for processing image data of an image sensor of the present application shown in the lower part, the image sensor scans its acquisition area and generates multiple image data in sequence. In the case where the scanning area of ​​the image sensor is pre-divided into three sub-acquisition areas A, B and C, the processing method of the present application starts to perform image processing when all image data generated by scanning the sub-acquisition area A is received (i.e., at time t11), that is, before all image data generated by scanning the entire acquisition area (i.e., at time t1) is received. This advances the processing flow from time t1 in the prior art to time t11 of the present application, thereby making the end time of image processing in the present application earlier than the end time of image processing in the existing solution, and thus the end time of the final algorithm processing will inevitably be advanced, so that the image recognition result can be obtained faster, and the end-to-end delay is ultimately shortened.

[0106] <Implementation scenario>

[0107] The method of processing image data of an image sensor of the present application can be applicable to processing image data generated by various "scanning" image sensors, and can be applied to vehicles, robots and other equipment having such image sensors.

[0108] For example, in a motion robot scene with a camera, the robot uses the camera to perceive the environment and plans and executes corresponding movements based on the image recognition results of the camera's image data. When such a robot applies the method of processing image sensor image data of the present application, it can obtain image recognition results more quickly, so as to make corresponding movements in response to the results more quickly. Therefore, the use of the method of processing image sensor image data of the present application can make the motion of the motion robot more agile.

[0109] For example, in a scenario of an intelligent networked vehicle with a laser radar, the vehicle uses the laser radar to sense the road conditions and plans and performs corresponding autonomous driving operations based on the image recognition results of the laser radar's point cloud image data. When such a robot applies the method of processing image sensor image data of this application, it can obtain image recognition results more quickly, thereby making corresponding driving operations in response to the results more quickly. Therefore, the use of the method of processing image sensor image data of this application can make the autonomous driving of intelligent networked vehicles safer.

[0110] In addition, although the method for processing image sensor image data of the present application is an improvement on the scheme for processing one frame of image, the time for processing one frame of image is shortened, thereby shortening the end-to-end delay. It should be understood that the method for processing image sensor image data of the present application is also applicable to processing videos including multiple frames of image. By applying the method for processing image sensor image data of the present application to each frame of image in the video, the processing time of the video can be shortened.

[0111] <Implementation Method>

[0112] The following will refer to Figures 2 to 9 An embodiment of an image sensor image data processing system and an embodiment of a method for processing image data of the present application are described in detail. Figure 3 In the figure, the optional module of the device is represented by a dotted frame, that is, the optional module can be omitted in some other implementations.

[0113] Figure 3 An image sensor image data processing system 1001 according to an embodiment of the present application is exemplarily shown, comprising an image data processing device 1100 and an image recognition device 1200 connected to each other.

[0114] Figure 3 The optional partition management module 1300 is shown in a dotted box. The partition management module 1300 is connected to the image data processing device 1100 and the image recognition device 1200 respectively, and is used to provide a partition strategy to the two. The partition strategy can be used to pre-set the size and number of multiple sub-acquisition areas of the acquisition area of ​​the image sensor. In this embodiment, the partition strategy provided by the partition management module 1300 can make the acquisition area be equally divided into three sub-acquisition areas A, B and C.

[0115] Figure 3 The image sensor 2000 is additionally shown. The image sensor 2000 scans the acquisition area and sequentially generates a plurality of first image data constituting a frame of image, and sequentially sends the plurality of first image data to the image data processing device 1100 in the image sensor image data transmission and processing system 1001 of the present application.

[0116] like Figure 4 As shown, the image data processing device 1100 includes a receiving module 1110 and an image processing module 1120 which are connected to each other.

[0117] In this implementation, the receiving module 1110 of the image data processing device 1100 is used to receive a plurality of first image data sent by the image sensor 2000 , and may also be used to receive a partitioning strategy provided by the partitioning management module 1300 .

[0118] In this embodiment, since the image processing module 1120 pre-divides the acquisition area into three equally sized sub-acquisition areas A, B and C according to the division strategy received by the receiving module 1110, the first image data received by the receiving module 1110 can be divided into three groups of first image data, so that the image sensor generates a group of first image data when scanning the three different sub-acquisition areas. The image processing module 1120 can thus be further used to perform image processing on a group of first image data when a group of first image data has been received.

[0119] Specifically, refer to Figure 2 , the image processing module 1120 is used to start performing image processing on the image data at time t11 when the receiving module 1110 receives all the first image data generated from the sub-acquisition area A. Then, at time t12, when the receiving module 1110 receives all the first image data generated from the sub-acquisition area B, the image processing module 1120 starts performing image processing on the image data. Finally, at time t13, when the receiving module 1110 receives all the first image data generated from the sub-acquisition area C, the image processing module 1120 starts performing image processing on the image data.

[0120] In the present application, the image processing module 1120 may include multiple image processing submodules (not shown). Similarly, image processing may include multiple image subprocessing. In practice, the image data processing device 1100 may be, for example, an image signal processor.

[0121] like Figure 5 As shown, the image recognition device 1200 includes a receiving module 1210, a feature extraction module 1220 and a fusion recognition module 1230 which are connected in sequence.

[0122] In this embodiment, the receiving module 1210 of the image recognition device 1200 is used to receive a plurality of second image data. The plurality of second image data is obtained by the image data processing device 1100 performing image processing on the plurality of first image data, and the plurality of first image data is sequentially generated by the image sensor 3000 scanning the acquisition area. In addition, the receiving module 1210 is also used to receive the partition strategy provided by the partition management module 1300.

[0123] In this embodiment, the feature extraction module 1220 is used to extract feature data from the received second image data in sequence during the period when the receiving module receives the plurality of second image data. Since the feature extraction module 1220 groups the second image data according to the division strategy received by the receiving module 1210, that is, the acquisition area is pre-divided into three sub-acquisition areas A, B, and C of equal size, the second image data received by the receiving module 1210 can also be similarly divided into three different groups of second image data, so that each second image data originating from the sub-acquisition areas A, B, and C belongs to a different group of second image data. The feature extraction module 1220 is used to extract feature data from a group of second image data when a group of second image data has been received.

[0124] Specifically, refer to Figure 2 , the feature extraction module 1220 is used to extract feature data from a group of second image data originating from the sub-acquisition area A when the receiving module 1210 receives the group of second image data. In other words, when all the first image data generated from the sub-acquisition area A are converted into corresponding second image data through image processing and received by the receiving module 1210, the feature extraction module 1220 starts to extract feature data from the second image data. Subsequently, the feature extraction module 1220 performs feature extraction on the second image data originating from the sub-acquisition area B and the sub-acquisition area C in a similar manner.

[0125] In this embodiment, the fusion recognition module 1230 is used to fuse all the feature data into a whole after the feature extraction module 1220 extracts feature data from all the second image data originating from the sub-acquisition areas A, B and C respectively, and then recognize the whole, and finally output the image recognition result at time t4.

[0126] Since both the feature extraction module 1220 and the fusion recognition module 1230 involve image recognition processing, in practice, they are usually set in the same entity, such as in the algorithm platform of the vehicle.

[0127] In this embodiment, the image processing module 1120 and the feature extraction module 1220 process the first image data generated from a sub-acquisition area or the second image data originated from a sub-acquisition area, respectively, that is, their processing objects are the first or second image data from a sub-acquisition area. The fusion recognition module 1230 of this embodiment is used to perform fusion recognition on all feature data from the second image data originated from the entire acquisition area as a whole, that is, its processing object is the feature data of all second image data originated from the entire acquisition area.

[0128] In addition, the feature extraction module 1220 may include one or more convolutional layers (not shown) and one or more pooling layers (not shown). The fusion identification module 1230 may include one or more feature fusion layers (not shown), one or more fully connected layers (not shown) and an output layer (not shown). These layers may also each include one or more sublayers (not shown). The various steps performed by various layers in the feature extraction module 1220 and the fusion identification module 1230 will be further described below, so the structures of the various layers in these modules will become clearer in the following related descriptions.

[0129] The following is combined with the above reference Figures 3 to 5 Description, reference Figure 6 An implementation of a method for processing image data of an image sensor according to the present application is described in detail.

[0130] Figure 6 The flowchart of the method for processing image data according to one embodiment of the present application is exemplarily shown, which includes the following steps S101 to S107:

[0131] In step S101, the partition management module 1300 provides a partition strategy. Specifically, the partition strategy is used to preset the number of sub-collection regions of a collection region to be 3 and to preset the sizes of the sub-collection regions to be equal to each other.

[0132] In step S102, the image data processing device 1100 and the image recognition device 1200 receive the division strategy respectively, and pre-set each sub-collection area of ​​the collection area according to the division strategy respectively, so as to group the first image data and the second image data respectively. Specifically, the receiving module 1110 of the image data processing device 1100 and the receiving module 1210 of the image recognition device 1200 receive the division strategy respectively, and the corresponding image processing module 1120 and the feature extraction module 1220 divide the collection area into the first sub-collection area and the second sub-collection area according to the division strategy respectively, and pre-set the first and second sub-collection areas to sub-collection areas A, B and C of equal size. In other words, since the division strategy adopted is the same, the second sub-collection areas of the image data processing device 1100 and the recognition device 1200 are both sub-collection areas A, B and C.

[0133] In step S103, the image data processing device 1100 receives a plurality of first image data sequentially generated by the image sensor scanning the acquisition area. Specifically, the receiving module 1110 of the image data processing device 1100 receives a plurality of first image data sequentially generated and sent by the image sensor 3000 scanning the acquisition area.

[0134] In step S104, when the image data processing device 1100 has received a group of first image data generated from a first sub-acquisition area, it performs image processing on the group of first image data and outputs corresponding second image data. Specifically, when the receiving module 1110 receives a group of first image data generated from a first sub-acquisition area, the image processing module 1120 performs image processing on the group of first image data and sequentially outputs corresponding second image data obtained by performing the image processing on the group of first image data.

[0135] In step S105, the image recognition device 1200 receives the plurality of second image data sequentially output by the image data processing device 1100. Specifically, the receiving module 1210 of the image recognition device 1200 receives the plurality of second image data sequentially output by the image data processing device 1100.

[0136] In step S106, when the image recognition device 1200 has received a group of second image data originating from a second sub-acquisition area, it extracts feature data from the group of second image data. Specifically, when the receiving module 1210 receives all the second image data originating from a second sub-acquisition area, the feature extraction module 1220 of the image recognition device 1200 extracts feature data from the image data together.

[0137] In step S107, the image recognition device 1200 performs fusion recognition processing on the feature data extracted from the plurality of second image data, and outputs an image recognition result. Specifically, after the feature extraction module 1220 of the image recognition device 1200 respectively completes the extraction of feature data from the second image data originating from the sub-collection areas A, B, and C, the fusion recognition module 1230 of the image recognition device 1200 fuses all the feature data into one, then recognizes the whole, and finally outputs the image recognition result.

[0138] It should be understood that the above steps S101 to S107 are not arranged in chronological order. For example, in this embodiment, when the receiving module 1110 in step S103 is receiving the first image data generated by the sub-acquisition area B, the image processing module 1120 in step S104 can perform image processing on the first image data generated by the sub-acquisition area A. Therefore, step S103 and step S104 can overlap in time. Similarly, step S105 and step S105 can overlap in time. In addition, in this embodiment, since the acquisition area is divided into three sub-acquisition areas, as the image sensor scans the sub-acquisition areas A, B and C in sequence, steps S104 and S106 are repeated three times accordingly to process the first image data generated from each sub-acquisition area and the second image data originating from each sub-acquisition area, respectively, as shown in FIG. Figure 2 shown.

[0139] In addition, it should be understood that the execution objects of the image processing step S104 and the feature extraction step S106 in this embodiment are the first or second image data generated or originated from a sub-acquisition area of ​​the acquisition area, and the execution object of the fusion recognition step S107 in this embodiment is all feature data extracted from the second image data originated from the entire acquisition area.

[0140] Reference again Figure 2 It can be known that the delay benefit ΔT achieved by the above system and method implementation compared with the existing solution can be calculated as follows:

[0141] ΔT=t3-t4 Formula (1);

[0142] Taking time t1 as the benchmark, equation (1) can be converted into:

[0143] ΔT=(t3-t1)-(t2-t1) / 3-T0 / 3-T1 Formula (2);

[0144] t3=t2+T0+T1, thus converting equation (2), we can get

[0145] ΔT = 2 / 3*(t2-t1+T0) Formula (3);

[0146] Among them, t1 is the starting time of image processing of the first image data generated from the entire acquisition area in the existing scheme, and t1 is also the starting time of image processing of the first image data generated from the sub-acquisition area C in this embodiment of the present application; t2 is the time when image processing ends and algorithm processing starts in the existing scheme; t3 is the time when algorithm processing ends in the existing scheme; t4 is the time when fusion recognition ends in this embodiment of the present application; T0 is the duration of feature extraction that is part of the algorithm processing in the existing scheme, and is also the duration of feature processing in this embodiment of the present application; and T1 is the duration of fusion recognition that is part of the algorithm processing in the existing scheme, and is also the duration of fusion recognition in this embodiment of the present application.

[0147] pass Figure 2 It can be clearly known from formula (3) that the delay benefit in this embodiment of the present application benefits from the advance of the image processing process, that is, in this embodiment, image processing of 2 / 3 of the first image data of a frame of image has been started before time t1, thereby reducing the end-to-end delay. The delay benefit in this embodiment of the present application benefits from the advance of the image recognition process, that is, in this embodiment, feature extraction of 2 / 3 of the second image data of the frame of image has been started before time t1, thereby further reducing the end-to-end delay.

[0148] The following reference Figure 7-9Some modules and steps in the system and method implementation of the present application are described in more detail.

[0149] Figure 7 The schematic diagram of dividing the acquisition area of ​​the image sensor 3000 using the division strategy provided by the division management module 1300 is exemplarily shown. Taking the camera as an example, Figure 7 As shown, a frame of image is generated by the camera exposing the target surface (also referred to as the acquisition area in the present invention) from top to bottom in rows. According to the division strategy provided by the division management module 1300, the acquisition area is divided into three rectangular sub-acquisition areas A, B and C from top to bottom, and the size of each sub-acquisition area is set to be equal.

[0150] exist Figure 7 In the example, the image sensor first scans the sub-acquisition area A and thus first generates the first image data of the sub-acquisition area A, and then first transmits the first image data generated from the sub-acquisition area A to the image sensor image data processing system 1001. The image sensor image data processing system 1001 therefore first performs image processing and feature extraction on the image data generated from the sub-acquisition area A, then processes the first image data generated from the sub-acquisition area B, and finally processes the first image data generated from the sub-acquisition area C. Thereafter, the acquisition area of ​​the camera can also be divided using the same division strategy to process the second frame image generated by subsequent scanning, until the Zth frame image, where Z is any integer greater than 2. In other words, the present application can process multiple frames of images, and therefore can process objects composed of multiple frames of images, such as videos.

[0151] Figure 8 yes Figure 7 Schematic diagram of the division of each sub-collection area of ​​the image sensor collection area. Assume that Figure 7 The resolution of the acquisition area shown is 1920*1080. The upper left corner of the acquisition area is set as the imaginary coordinate origin, and each sub-acquisition area A, B and C can be easily defined by the coordinates of the upper left corner, lower left corner, upper right corner and lower right corner of the rectangle. For specific coordinates, see Table 1 and Figure 8 .

[0152] Table 1

[0153] Coordinates of the upper left corner Coordinates of the upper right corner Coordinates of the lower left corner Coordinates of the lower right corner Sub-collection area A (0,0) (1919,0) (0,359) (1919,359) Sub-collection area B (0,360) (1919,360) (0,719) (1919,719) Sub-collection area C (0,720) (1919,720) (0,1079) (1919,1079)

[0154] In this way, each sub-collection area of ​​the collection area can be defined in a simple and flexible manner, and the number of sub-collection areas and the size of each sub-collection area can be adjusted according to actual conditions.

[0155] Reference below Fig. 9The feature extraction step S106 and the fusion recognition step S107 as well as the structures of the related step execution volume feature extraction module 1220 and the fusion recognition module 1230 are further described.

[0156] Fig. 9 The schematic diagram of the feature extraction and fusion recognition process in the method for processing image data according to one embodiment of the present application is exemplarily shown. The parts involving the feature extraction step S106 and the fusion recognition step S107 are marked with different dotted boxes, respectively. Among them, conv1 indicates the first convolution processing that can be performed by the first convolution layer of the feature extraction module 1220, conv2 indicates the second convolution processing that can be performed by the second convolution layer of the feature extraction module, and conv3 indicates the third convolution processing that can be performed by the third convolution layer of the feature extraction module; similarly, pool1 to pool3 each indicate the first to third pooling processing that can be performed by the first to third pooling layers of the feature extraction module 1220, respectively, and fc1 and fc2 each indicate the first and second fully connected processing that can be performed by the first and second fully connected layers of the fusion recognition module 1230, respectively. concat indicates a series of feature fusion processing that can be performed by a series of feature fusion layers of the fusion recognition module 1230. output indicates the output of the image recognition result that can be performed by the output layer of the fusion recognition module 1230. In addition, in Fig. 9 , conv2_1 indicates a first subconvolution process of a first convolution process that can be performed by a first subconvolution layer of a first convolution layer, and conv2_2 indicates a second subconvolution process of a first convolution process that can be performed by a second subconvolution layer of the first convolution layer. Similarly, conv3_1 to conv3_3 each indicate first to third subconvolution processes of a third convolution process that can be performed by first to third subconvolution layers of a third convolution layer, respectively.

[0157] For ease of display, Fig.10 There is no process involved in step S106 arranged in chronological order. Specifically, when the feature extraction module 1220 executes the feature extraction step S106, as shown in the dotted box of S106, first, the various convolutions and pooling processes of the image data of the sub-acquisition area A in the left column are executed from top to bottom, and then the various convolutions and pooling processes of the image data of the sub-acquisition area B in the middle column are executed from top to bottom, and finally, the various convolutions and pooling processes of the image data of the sub-acquisition area C in the right column are executed from top to bottom. In other words, the convolution and pooling of the second image data of each sub-acquisition area A, B and C are not performed simultaneously, but are performed sequentially from left to right in units of the second image data of a sub-acquisition area.

[0158] like Fig.10As shown in the dotted box of S107, when the fusion recognition module 1230 executes the fusion recognition step S107, a series of feature fusion processing, a first full connection processing, a second full connection processing and the output of the image recognition result are performed in sequence from top to bottom.

[0159] For ease of understanding, an embodiment of the image sensor image data processing system and an embodiment of the image data processing method of the present application are described in detail above. However, it should not be understood that the system and method of the present application are limited to the feature combinations of the above embodiments.

[0160] In some other embodiments of the present application, the image data processing device 1100 takes a set of first image data generated from a sub-acquisition area as a processing object, and the image recognition device 1200 takes all second image data from the entire acquisition area as a processing object. Fig.10 As shown in the lower part, the method for image data processing of this embodiment of the present application includes: during the reception of multiple first image data generated from scanning the entire acquisition area, sequentially performing image processing on a group of first image data generated from a sub-acquisition area to output corresponding second image data; after completing image processing on all first image data generated from three sub-acquisition areas and outputting all second image data, performing feature extraction and fusion recognition on all second image data as a whole. Therefore, the delay benefit of this solution is only 2 / 3 of the image processing time, that is, this solution only benefits from the advance of image processing of the first image data generated from sub-acquisition areas A and B.

[0161] In some other embodiments, the sizes of the sub-collection areas may be defined to be unequal to each other. Fig.11 In another embodiment of the present application shown, the size of the sub-collection area D is set to be 1 / 2 of the size of the sub-collection area E.

[0162] In some other embodiments, the image data processing device or image recognition device may start processing or extracting features from the received first or second image data at a time other than the time when the image data is received. Fig.11In another embodiment of the present application shown in the lower part, the image data processing device performs image processing on the first image data generated from the sub-acquisition area D at time t16, that is, between time t14 when all the first image data generated from the sub-acquisition area D are received and time t15 when all the first image data generated from the sub-acquisition area E are received. In this way, the image data processing device starts performing image processing on all the first image data generated from the sub-acquisition area E at the same time as the completion time of performing image processing on all the first image data generated from the sub-acquisition area D. Therefore, the delay benefit of this solution is only 1 / 3 of the image processing time, that is, this solution only benefits from the advance of image processing of all the first image data generated from the sub-acquisition area D.

[0163] Therefore, refer to Figure 2 , Fig.10 and Fig.11 It can be understood that the setting of the start time of the image data processing device performing image processing and the image recognition device performing feature extraction will affect the delay benefit of the present application. In addition, the setting of the number and size of the sub-collection areas of the collection area will also affect the delay benefit of the present application.

[0164] In some other implementations, the size and number of the sub-collection areas of the collection area can be directly pre-configured according to preset values, so the division management module 1300 can be omitted, or steps S101 and S102 can be omitted. Fig.10 In the scheme shown in the lower part, the image data processing device 1100 and the image recognition device 1200 perform the following steps S201-S205 according to the preset values. Fig.12 shown.

[0165] In step S201, the image data processing device 1101 receives a plurality of first image data sequentially generated by an image sensor scanning a collection area. Specifically, the receiving module 1111 of the image data processing device 1101 receives a plurality of first image data sequentially generated by an image sensor scanning a collection area.

[0166] In step S202, the image data processing device 1101 performs image processing on the group of first image data when it has received a group of first image data, and the group of first image data is generated from a preset first sub-acquisition area of ​​the acquisition area. Specifically, according to a predetermined value, the image processing module 1121 of the image data processing device 1100 pre-sets each of the first sub-acquisition areas A, B and C of the acquisition area. When the receiving module 1111 receives a group of first image data generated from a first sub-acquisition area, the image processing module 1121 of the image data processing device 1100 performs image processing on the first image data, and sequentially outputs each of the corresponding second image data obtained by performing the image processing on the group of first image data.

[0167] In step S203 , the image recognition device 1201 receives a plurality of second image data sequentially output by the image data processing device 1101 . Specifically, the second image data are received by the receiving module 1211 of the image recognition device 1201 .

[0168] In step S204, when the image recognition device 1201 receives the plurality of second image data, it extracts feature data from the plurality of second image data. Specifically, when the receiving module 1211 receives the plurality of second image data from the entire acquisition area, the feature extraction module 1221 of the image recognition device 1201 extracts feature data from the plurality of second image data together.

[0169] In step S205, the image recognition device 1201 performs fusion recognition processing on the feature data extracted from the plurality of second image data, and outputs an image recognition result. Specifically, the feature extraction module 1221 of the image recognition device 1201 extracts feature data from the plurality of second image data, and the fusion recognition module 1231 of the image recognition device 1201 fuses the plurality of feature data thus obtained into one, and then performs recognition, and finally outputs the image recognition result.

[0170] In some other embodiments, the first image data and the second image data of different frames may be grouped in different ways, for example, the number and size of the sub-collection areas of the collection area when the camera scans different frames of images may be pre-set using different division strategies or preset values. In one such embodiment, in processing the image data of the first frame of image, the image data of the second frame of image may be grouped using Figure 7 The division method shown in FIG. 1 divides the acquisition area into three sub-acquisition areas of equal size. In processing the image data of the second frame image, the image data of the second frame image can be processed by using Fig.11 The adopted division strategy is to divide the collection area into two sub-collection areas of unequal sizes, wherein the size of one sub-collection area is twice that of the other.

[0171] In some other embodiments, the first image data and the second image data of the same frame may be grouped in different ways, for example, in the image data processing process, each sub-acquisition area of ​​the acquisition area is set in one way, and in the image recognition process, each sub-acquisition area of ​​the acquisition area is set in another way. In one such embodiment, in the image data processing process, the acquisition area is divided into four equal sub-acquisition areas F, G, H and I, and in the image recognition process, the acquisition area is divided into two equal sub-acquisition areas J and K, wherein sub-acquisition area J coincides with the intersection of sub-acquisition areas F and G, and sub-acquisition area K coincides with the intersection of sub-acquisition areas H and I. In this way, the image data processing process can be advanced, and the image recognition process can be ensured not to be overly complicated, thereby avoiding increasing the time of the image recognition process.

[0172] In some other embodiments, especially those in which the image sensor is a lidar, each sub-capture area of ​​the image may be defined not by the coordinates of the four corners but by the rotation angle. For example, a sub-capture area of ​​the point cloud may be defined by a rotation angle range of 0° to 90°.

[0173] In some other embodiments, the number of each layer and each sub-layer of the feature extraction module and the fusion recognition module is adjustable. Accordingly, in some other embodiments, the number of times each process and each sub-process of feature extraction and fusion recognition is performed is also adjustable.

[0174] In some embodiments, the image data processing device receives first image data from an image sensor, the first image data being one of a plurality of image data that can be generated by the image sensor scanning a physical area corresponding to a collection area in one scanning cycle, the collection area representing the collection range of the image sensor. The image data processing device performs image processing on the first image data to obtain second image data. The image data processing device outputs the second image data. That is, in this embodiment, in the image processing process, the first image data is not processed in a grouping manner.

[0175] In some embodiments, the image recognition device sequentially receives second image data, which is obtained by performing image processing on the first image data, and the first image data is one of multiple image data that can be generated by the image sensor scanning the physical area corresponding to the acquisition area in one scanning cycle, and the acquisition area represents the acquisition range of the image sensor. The image recognition device sequentially extracts feature data from the second image data. The image recognition device performs fusion recognition processing on each feature data. The image recognition device outputs an image recognition result. That is, in this embodiment, in the image recognition process, the second image data is not processed in a grouping manner.

[0176] Fig.13 The schematic diagram of the structure of a driving system 3001 according to one embodiment of the present application is exemplarily shown. The driving system 3001 is an advanced driver assistance system (ADAS), which includes an image sensor image data processing system 1001 and a driving decision unit 3100. The image sensor image data processing system 1001 can be connected in communication with a camera 2001 outside the driving system 3001, process and recognize a plurality of first image data sequentially generated by scanning the acquisition area by the camera 2001 and output the image recognition result. The driving decision unit 3100 is connected in communication with the image sensor image data processing system 1001, and is used to perform behavior decision and motion planning according to the image recognition result output by the image sensor image data processing system 1001 and output operation instructions.

[0177] Fig.14 The schematic diagram of the structure of an intelligent networked vehicle V according to an embodiment of the present application is exemplarily shown. The intelligent networked vehicle V includes a camera 2001 usually set in front of the vehicle, a driving system 3001 set in the vehicle, an electronic control unit 4001 and an actuator 5001 such as a braking mechanism. The camera 2001 senses the vehicle environment by scanning its acquisition area in rows and outputs a plurality of first image data in sequence. The driving system 3001 is connected in communication with the camera 2001, and is used to output operation instructions according to the plurality of first image data from the camera 2001. The electronic control unit (Electronic Control Unit, ECU) 4001 is connected in communication with the driving system 3001, and is used to control the actuator 5001 to perform operations according to the operation instructions from the driving system 3001, for example, controlling the braking mechanism to perform a braking operation according to the braking instructions of the driving system.

[0178] Fig.15 15 is a structural exemplary diagram of a computing device 1500 provided in an embodiment of the present application. The computing device 1500 includes: a processor 1510 , a memory 1520 , a communication interface 1530 , and a bus 1540 .

[0179] It should be understood that Fig.15 The communication interface 1530 in the computing device 1500 shown can be used to perform communications with other devices.

[0180] The processor 1510 may be connected to a memory 1520. The memory 1520 may be used to store the program code and data. Therefore, the memory 1520 may be a storage unit inside the processor 1510, or an external storage unit independent of the processor 1510, or a component including a storage unit inside the processor 1510 and an external storage unit independent of the processor 1510.

[0181] Optionally, the computing device 1500 may further include a bus 1540. The memory 1520 and the communication interface 1530 may be connected to the processor 1510 via the bus 1540. The bus 1540 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus 1540 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.

[0182] It should be understood that in the implementation of the present application, the processor 1510 can use a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. Or the processor 1510 uses one or more integrated circuits to execute related programs to implement the technical solutions provided in the implementation of the present application.

[0183] The memory 1520 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1510. A portion of the processor 1510 may also include a nonvolatile random access memory. For example, the processor 1510 may also store information on the device type.

[0184] When the computing device 1500 is running, the processor 1510 executes the computer-executable instructions in the memory 1520 to perform the operating steps of any of the above-mentioned methods for processing image data of an image sensor.

[0185] In some other implementations, the communication interface 1530 and the bus 1540 are omitted.

[0186] It should be understood that the computing device 1500 according to the implementation mode of the present application can correspond to the corresponding subject in executing the method according to each implementation mode of the present application, and the above-mentioned and other operations and / or functions of each unit in the computing device 1500 are respectively for realizing the corresponding process of each method of the present implementation mode, which will not be described in detail here for the sake of brevity.

[0187] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0188] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only exemplary. For example, the division of the units is only a logical function division. There may be other division strategies in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0192] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, it is used to execute a method for processing image data of an image sensor, which method includes at least one of the methods described in the above embodiments.

[0194] The computer storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive lists) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, clouds (cloud) or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by instruction execution systems, devices or devices or used in combination with them.

[0195] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0196] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0197] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0198] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the present application, all of which belong to the scope of protection of the present application.

Claims

1. A method for processing image data of an image sensor, characterized in that: include: The partition management module provides a partition strategy, wherein the partition strategy is used to preset the number of sub-collection areas of the collection area and to preset the size of each sub-collection area; The image data processing device and the image recognition device respectively receive the division strategy; The image data processing device pre-sets each sub-acquisition area of ​​the acquisition area according to the division strategy to divide the acquisition area into a plurality of first sub-acquisition areas; The image recognition device pre-sets each sub-collection area of ​​the collection area according to the division strategy to divide the collection area into a plurality of second sub-collection areas; The image data processing device receives a plurality of first image data sequentially generated from a scanning acquisition area of ​​an image sensor; The image data processing device, when having received a group of first image data generated from a first sub-acquisition area, performs image processing on the group of first image data and sequentially outputs second image data obtained by performing image processing on the group of first image data; The image recognition device receives a plurality of second image data sequentially output by the image data processing device; The image recognition device extracts feature data from a set of second image data when receiving the set of second image data originating from a second sub-capturing area; The image recognition device performs fusion recognition processing on feature data extracted from second image data corresponding to all second sub-collection areas included in the collection area, and outputs an image recognition result.

2. A sensor image data processing system, characterized in that: The sensor image data processing system comprises: an image data processing device, an image recognition device and a division management module, wherein the image data processing device comprises a first receiving module and an image processing module, and the image recognition device comprises a second receiving module, a feature extraction module and a fusion recognition module, wherein: The partition management module is used to provide a partition strategy, wherein the partition strategy is used to pre-set the number of sub-collection areas of a collection area and to pre-set the size of each sub-collection area; The first receiving module is used to receive the division strategy and to receive a plurality of first image data sequentially generated from the image sensor scanning and collecting area; The image processing module is used to pre-set the sub-collection areas of the collection area according to the division strategy to divide the collection area into a plurality of first sub-collection areas and to perform image processing on a group of first image data generated from a first sub-collection area when the first receiving module has received the group of first image data and sequentially output second image data obtained by performing image processing on the group of first image data; The second receiving module is used to receive the division strategy and to receive a plurality of second image data sequentially output by the image data processing device; The feature extraction module is used to pre-set each sub-collection area of ​​the collection area according to the division strategy to divide the collection area into a plurality of second sub-collection areas and to extract feature data from a group of second image data from a second sub-collection area when the second receiving module has received the group of second image data; and The fusion recognition module is used to perform fusion recognition processing on feature data extracted from the second image data corresponding to all second sub-collection areas included in the collection area, and output an image recognition result.

3. A driving system, characterized in that: comprising the sensor image data processing system and the driving decision unit as claimed in claim 2; The driving decision unit is connected to the sensor image data processing system, and is used to execute behavior decisions and motion planning according to the image recognition results output by the sensor image data processing system, and output operation instructions.

4. A vehicle, characterized in that: The invention comprises an image sensor, a driving system as claimed in claim 3, an electronic control unit and an actuator connected in sequence; wherein The image sensor is used to sense the vehicle environment in a scanning manner and output first image data; The electronic control unit is used to control the actuator to perform operations according to the operation instructions of the driving system.

5. A computing device, characterized in that: include: at least one processor; as well as At least one memory is connected to the processor and stores program instructions, which, when executed by the at least one processor, cause the at least one processor to perform the method of claim 1.

6. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a computer, the computer is caused to perform the method of claim 1 .

Images