Image frame processing chip and vehicle

Abstract

The application discloses an image frame processing chip and a vehicle. A storage module in the image frame processing chip can store multiple offset point information and attention weights corresponding to a reference point of a feature map of an image frame. An acquisition module can send a data reading request to the storage module and receive the multiple offset point information and the attention weights sent by the storage module. A determination module can determine an attention processing result of the feature map according to the multiple offset point information and the attention weights forwarded by the acquisition module, and store the attention processing result in the storage module to complete the processing of the image frame. In this way, the determination of the attention processing result can be realized through one data reading and writing step, thereby reducing the reading and writing overhead caused by the data reading step in the determination process of the attention processing result, and further weakening the negative influence caused by the attention mechanism in the calculation process of the neural network model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image frame processing chip and a vehicle. BACKGROUND

[0002] With the continuous development of AI (Artificial Intelligence) technology, attention mechanisms are widely used in natural language processing, computer vision and other fields. However, although the attention mechanism can enable the neural network model to focus on specific data information to improve the prediction accuracy of the model, it also introduces corresponding overhead, which to some extent affects the computing efficiency of the model. SUMMARY

[0003] The present application provides an image frame processing chip and a vehicle.

[0004] The image frame processing chip provided by the present application embodiment comprises:

[0005] The storage module is configured to store a plurality of offset point information and attention weights corresponding to a reference point of a feature map of an image frame;

[0006] The acquisition module is configured to send a data reading request to the storage module, and receive the plurality of offset point information and the attention weights sent by the storage module;

[0007] The determination module is configured to determine an attention processing result of the feature map according to the plurality of offset point information and the attention weights forwarded by the acquisition module, and store the attention processing result into the storage module to complete the processing of the image frame.

[0008] In the image frame processing chip provided by the present application embodiment, the storage module can store a plurality of offset point information and attention weights corresponding to a reference point of a feature map of an image frame, the acquisition module can send a data reading request to the storage module, and receive the plurality of offset point information and the attention weights sent by the storage module, and the determination module can determine an attention processing result of the feature map according to the plurality of offset point information and the attention weights forwarded by the acquisition module, and store the attention processing result into the storage module to complete the processing of the image frame.

[0009] In this way, in the present application embodiment, the determination module of the image frame processing chip can determine the attention processing result of the feature map of the image frame based on the offset point information and the attention weights obtained by the acquisition module from the storage module, so that the determination of the attention processing result can be realized through one data reading and writing step, thereby reducing the reading and writing overhead caused by the data reading step in the determination process of the attention processing result, and further weakening the negative influence caused by the attention mechanism in the computing process of the neural network model.

[0010] In some embodiments of the present application, the plurality of offset point information corresponding to the reference point comprises coordinate information of the plurality of offset points and interpolation pixel values corresponding to the plurality of offset points, and the determining module comprises a first determining submodule and a second determining submodule.

[0011] The first determining submodule is configured to determine target weights corresponding to the offset points according to the attention weights forwarded by the obtaining module and the coordinate information of the offset points.

[0012] The second determining submodule is configured to determine the attention processing result according to the target weights sent by the first determining submodule and the interpolation pixel values forwarded by the obtaining module.

[0013] In this way, in the embodiments of the present application, target weights can be determined based on attention weights, coordinate information of offset points of a feature map, and interpolation pixel values corresponding to the offset points, and the attention processing result can be determined according to the target weights and pixel values of the interpolation pixel points, thereby realizing online calculation of the attention processing result.

[0014] In some embodiments of the present application, the coordinate information of the offset points comprises a decimal part of the coordinate of the offset point, and the first determining submodule is configured to:

[0015] determine an interpolation pixel coefficient corresponding to the offset point according to the decimal part of the offset point;

[0016] determine the target weight corresponding to the offset point according to the attention weight and the interpolation pixel coefficient corresponding to the offset point.

[0017] In this way, in the embodiments of the present application, the determination of the interpolation pixel coefficient can be completed based on the decimal part of the coordinate of the offset point, thereby reducing the overhead caused by data interaction in the process of calculating the attention processing result to a certain extent and enabling the interpolation pixel coefficient to be determined efficiently.

[0018] In some embodiments of the present application, the second determining submodule is configured to:

[0019] determine a sub-attention processing result corresponding to each of the offset points according to the target weight corresponding to each of the offset points and the interpolation pixel value;

[0020] determine the attention processing result according to the sub-attention processing result corresponding to each of the offset points.

[0021] In this way, in the embodiments of the present application, the attention processing result of a feature map can be determined according to the sub-attention processing result of each offset point in the feature map.

[0022] In some embodiments of the present application, the offset point information corresponding to the reference point comprises coordinate information of the offset points and interpolation pixel values corresponding to the offset points, and the determining module comprises a first determining submodule and a second determining submodule.

[0023] The first determining submodule is configured to determine interpolation pixel coefficients corresponding to the offset points according to the coordinate information of the offset points forwarded by the obtaining module, construct a coefficient matrix according to the interpolation pixel coefficients corresponding to each offset point, and determine a weight matrix according to the coefficient matrix and the attention weight.

[0024] The second determining submodule is configured to determine the attention processing result according to the weight matrix sent by the first determining submodule and the interpolation pixel values corresponding to the offset points forwarded by the obtaining module.

[0025] In this way, in the embodiments of the present application, the attention processing result of the feature map can be determined according to the sub-attention processing result of each offset point in the feature map.

[0026] In some embodiments of the present application, the offset point information corresponding to the reference point comprises coordinate information of the offset points and interpolation pixel values corresponding to the offset points, and the determining module comprises a first determining submodule and a second determining submodule.

[0027] The first determining submodule is configured to determine interpolation pixel coefficients corresponding to the offset points according to the coordinate information of the offset points forwarded by the obtaining module, and determine offset point pixel values of the offset points according to the interpolation pixel coefficients corresponding to the offset points and the interpolation pixel values corresponding to the offset points forwarded by the obtaining module.

[0028] The second determining submodule is configured to determine the attention processing result according to the attention weight forwarded by the obtaining module and the offset point pixel values sent by the first determining submodule.

[0029] In this way, in the embodiments of the present application, the interpolation pixel coefficients of the interpolation pixel points corresponding to the offset points can be determined according to the coordinate information of the offset points, a coefficient matrix can be constructed according to the interpolation pixel coefficients of the interpolation pixel points corresponding to each offset point, and the determination of the attention processing result can be completed in combination with the matrix operation mode according to the coefficient matrix, the attention weight and the interpolation pixel values, so that the attention processing result can be efficiently determined.

[0030] In some embodiments of the present application, the offset point information corresponding to the reference point comprises coordinate information of the offset points and interpolation pixel values corresponding to the offset points, and the determining module comprises a first determining submodule and a second determining submodule.

[0031] The first determining submodule is configured to determine a pixel weighted result corresponding to the offset point according to the interpolation pixel value corresponding to the offset point and the attention weight forwarded by the obtaining module.

[0032] The second determining submodule is configured to determine the attention processing result according to the pixel weighted result corresponding to the offset point sent by the first determining submodule and an interpolation pixel coefficient determined by the coordinate information forwarded by the obtaining module.

[0033] In this way, in the embodiments of the present application, the attention processing result of the feature map can be determined according to the attention weight of the feature map and the pixel value of the offset point in the feature map, so that the attention processing result of the feature map can be efficiently determined.

[0034] In some embodiments of the present application, the offset point information includes offset point pixel values of the plurality of offset points, and the determining module is configured to:

[0035] determine a sub-attention processing result corresponding to the offset point according to the offset point pixel value and the attention weight forwarded by the obtaining module;

[0036] determine the attention processing result according to the sub-attention processing results corresponding to the plurality of offset points.

[0037] In this way, in the embodiments of the present application, the attention weight, the coordinate information of the offset point of the feature map, and the pixel value of the interpolation pixel point corresponding to the offset point can be obtained to determine the offset point pixel value, and the attention processing result can be determined according to the offset point pixel value and the attention weight, thereby realizing online calculation of the attention processing result.

[0038] In some embodiments of the present application, the coordinate information of the plurality of offset points corresponding to the reference point is obtained based on a neural network model that has been pre-trained and pixel random selection on the feature map.

[0039] In this way, in the embodiments of the present application, the coordinate information of the plurality of offset points can be determined based on the neural network model and pixel random selection.

[0040] The embodiments of the present application provide a vehicle including the image frame processing chip.

[0041] The vehicle provided by the embodiment of the present application can determine the attention processing result of the feature map of the image frame based on the offset point information and the attention weight obtained by the acquisition module from the storage module, so that the determination of the attention processing result can be realized by one data reading and writing step, thereby reducing the reading and writing overhead caused by the data reading step in the determination process of the attention processing result, and further weakening the negative influence caused by the attention mechanism in the calculation process of the neural network model.

[0042] Additional aspects and advantages of the embodiments of the present application will be in part apparent and in part pointed out hereinafter in the description of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the description of the embodiments, taken in conjunction with the following drawings in which:

[0044] Figure 1 A schematic diagram of an image frame processing chip in some embodiments of the present application;

[0045] Figure 2 A schematic diagram of an image frame processing chip in some embodiments of the present application;

[0046] Figure 3 A schematic diagram of an image frame processing chip in some embodiments of the present application;

[0047] Figure 4 A schematic diagram of an application scenario in some embodiments of the present application;

[0048] Figure 5 A schematic diagram of an image frame processing chip in some embodiments of the present application;

[0049] Figure 6 A schematic diagram of an image frame processing chip in some embodiments of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0051] With the continuous development of AI (Artificial Intelligence), the attention mechanism is more and more concerned by the industry and is widely used in natural language processing, computer vision and other fields.

[0052] Meanwhile, with the application of attention mechanism, the variants of attention mechanism also increase, such as channel attention, space attention, self-attention and deformable attention. Among them, due to the advantages of low computational complexity and adaptability to non-fixed positions, deformable attention has been widely used in the visual field with large amount of calculation and the need for separate design of visual targets.

[0053] However, in the actual application of deformable attention, GPU (Graphics Processing Unit) and / or CPU (Center Processing Unit) are usually used to linearly complete each calculation step of deformable attention. Among them, the calculation formula of deformable attention can be referred to as follows, that is:

[0054]

[0055] In the formula, q represents the query vector index, z q represents the input query feature vector, p q represents the offset reference point of the query vector, x represents the input feature map, m represents the attention head index, M represents the total number of attention heads, k represents the sampling point index, K represents the total number of sampling points, A mqk represents the attention weight of the query feature vector q at the kth offset point of the mth attention head, W m represents the output mapping matrix, W m ' represents the input mapping matrix of x, Δp mqk represents the offset of the query feature vector q at the kth offset point of the mth attention head.

[0056] Further, in the implementation scheme of the traditional deformable attention mechanism, for each offset point of the sampling point, p q , Δp mqk and W m '(or p q + Δp mqk ) are usually read from the external memory, the corresponding multiplication and / or addition operation is completed, the operation result is written to the external memory, then the operation result and x are read from the external memory, and the corresponding multiplication operation is performed, thereby obtaining the pixel operation result of the sampling point. Then the pixel operation result is written to the external memory, and the pixel operation result is read from the external memory and A mqkand the pixel operation result and A mqk corresponding product operation is performed, thereby obtaining the sampling result corresponding to the sampling point. Finally, according to the sampling results corresponding to the K sampling points of the M attention heads, the deformable attention calculation result is obtained.

[0057] It can be understood that in this linear calculation process, multiple external memory read and write operations are involved. The read and write operations of the external memory usually need to consider queuing, priority, preemption and the like, and the overhead of the read and write operations of the external memory is large, thereby resulting in a large amount of resources required for the calculation of the deformable attention and a long time required for the calculation, which finally affects the calculation efficiency of the neural network model.

[0058] It can also be understood that in a vehicle-mounted system image processing scene with high real-time requirement, such as an automatic driving scene, the resources required for the calculation of the deformable attention and the time required for the calculation may affect the calculation efficiency of the vehicle-mounted image processing model, and thus the automatic driving function of the vehicle is affected.

[0059] Based on the above possible problems, please refer to Figure 1 The embodiment of the present application provides an image frame processing chip 100, which comprises:

[0060] The storage module 110 is configured to store a plurality of offset point information and attention weights corresponding to reference points of a feature map of an image frame.

[0061] The acquisition module 120 is configured to send a data reading request to the storage module 110, and receive the plurality of offset point information and the attention weights sent by the storage module 110.

[0062] The determination module 130 is configured to determine an attention processing result of the feature map according to the plurality of offset point information and the attention weights forwarded by the acquisition module 120, and store the attention processing result into the storage module 110, thereby completing the processing of the image frame.

[0063] Specifically, in the embodiment of the present application, the vehicle can be equipped with or installed with the image frame processing chip 100, and thus in the process of processing the feature map of the image frame by the vehicle using the neural network model, the storage module 110 of the image frame processing chip 100 can store a plurality of offset point information and attention weights corresponding to reference points of the feature map of the image frame, and the acquisition module 120 of the image frame processing chip 100 can send a data reading request to the storage module 110, so that the storage module 110 sends the plurality of offset point information and the attention weights corresponding to the reference points of the feature map to the acquisition module 120.

[0064] Further, in the case that the acquisition module 120 receives the multiple offset point information corresponding to the reference point of the feature map and the attention weight, the acquisition module 120 can forward the multiple offset point information corresponding to the reference point of the feature map and the attention weight to the determination module 130. The determination module 130 can determine the attention processing result of the feature map according to the received multiple offset point information corresponding to the reference point of the feature map and the attention weight.

[0065] It can be understood that, in the embodiments of the present application, when the vehicle control camera component performs a shooting action to obtain an image frame, the feature extraction processing on the image frame can obtain a feature map.

[0066] Further, the offset point in the embodiments of the present application can be understood as a point determined by the offset and the reference point coordinates in the feature map. In addition, the attention weight can be understood as the weight corresponding to different offset points, which can be used to calculate the weighted value of the pixel value of the offset point.

[0067] It can also be understood that the reference point, the offset, the offset point and the attention weight in the embodiments of the present application can be understood as data that can be determined by a pre-trained neural network model, or in other words, these data can be given by a pre-trained neural network model.

[0068] Optionally, in some embodiments of the present application, the reference point, the offset, the offset point and the attention weight can be represented by the following formula, that is:

[0069]

[0070] In the formula, DeformAttn() represents deformable attention, q represents query vector index, z q represents input query feature vector, p q represents offset reference point of query vector, x represents input feature map, m represents attention head index, M represents total number of attention heads, k represents sampling point index, K represents total number of sampling points, A mqk represents attention weight of the query feature vector q at the kth offset point on the mth attention head, W m represents output mapping matrix, W m represents input mapping matrix of x, Δp mqk represents offset of the query feature vector q at the kth offset point on the mth attention head.

[0071] As shown in the above formula, p q can be understood as the reference point of the embodiments of the present application, Δp mqk can be understood as the offset described above, (p q + Δp mqk ) can be understood as the offset point of the embodiments of the present application, Amqk The attention weight can be understood as the attention mechanism of the embodiments of the present application.

[0072] It can be understood that when the attention mechanism is calculated by the above formula, the specific values of the parameters in the formula, such as A mqk , Δp mqk , W m ' and p q , can be given by a pre-trained neural network model.

[0073] It can also be understood that the offset point information in the embodiments of the present application can be understood as data related to p q in the above formula, such as W m 'x(p q + Δp mqk ). It can be understood that W m 'x(p q + Δp mqk ) can represent the pixel value of the offset point in the feature map.

[0074] It should be noted that in the embodiments of the present application, the offset point information and the attention weight are known quantities pre-stored in the memory. In order to more clearly illustrate the embodiments of the present application, please refer to Figure 2 , Figure 2 is a schematic diagram of an image frame processing chip 100 in some embodiments of the present application. That is, the acquisition module 120 of the embodiments of the present application can read the offset point information and the attention weight from the storage module 110, and forward the offset point information and the attention weight to the determination module 130. The determination module 130 can determine the attention processing result of the feature map according to the offset point information and the attention weight, thereby completing the calculation of the deformable attention.

[0075] It can be understood that the embodiments of the present application can read all the data required for calculating the attention processing result, and complete the calculation of the attention processing result in an online manner, that is, after reading the offset point information and the attention weight required for calculating the attention processing result, the corresponding calculation unit (such as the determination module 130) is used to calculate the attention processing result through the offset point information and the attention weight, and the intermediate results (such as the pixel operation results mentioned above) in the calculation process are no longer written into the corresponding storage unit (such as the storage module 110), but are directly used for the next calculation step, until the final calculation result is obtained, that is, the attention processing result is obtained, and then the calculation unit (such as the determination module 130) writes the attention processing result into the storage unit (such as the storage module 110) so that the corresponding device (such as an external processing module) reads the attention processing result, and captures the feature elements or feature information in the feature map according to the attention processing result.

[0076] It can also be understood that, based on Figure 1 or Figure 2 The implementation of the attention processing result calculation method, the embodiments of the present application enable the intermediate calculation result to be no longer written to the storage unit (such as the storage module 110) in the complete calculation process of the attention processing result, so that the complete calculation process includes one read and write of the storage unit (such as the storage module 110).

[0077] Therefore, compared with the multiple storage unit read and write included in the traditional deformable attention mechanism implementation scheme, the embodiments of the present application can reduce the read and write times of the storage unit, thereby reducing the read and write overhead of the storage unit, and thus can reduce the resources and time required for the calculation of deformable attention, so that the negative impact of the deformable attention mechanism on the model calculation efficiency can be alleviated for the neural network model implementing the deformable attention mechanism.

[0078] In summary, in the embodiments of the present application, the determination module 130 of the image frame processing chip 100 can determine the attention processing result of the feature map of the image frame based on the offset point information and the attention weight required for the attention processing result calculation obtained by the acquisition module 120 from the storage module 110, so that the determination of the attention processing result can be realized by one data read and write step, thereby reducing the read and write overhead caused by the data read step in the attention processing result determination process, and thus the negative impact caused by the attention mechanism in the calculation process of the neural network model can be weakened.

[0079] Moreover, the access and read and write of the memory in the image processing process are reduced, so that the access conflict existing when the memory is shared with other processing units of the vehicle can be reduced to a certain extent, and thus the negative impact of the image processing on the performance of the vehicle-mounted system can be reduced.

[0080] In addition, in the vehicle-mounted system, the access of the memory can be based on a bus. Furthermore, since the access and read and write of the memory in the image processing process are reduced, the power consumption of the bus and the memory can be reduced.

[0081] In addition, since the embodiments of the present application read the "various data required for determining the attention processing result" only once, the data read step is reduced, and thus the calculation time caused by the data read step is reduced, thereby improving the efficiency of the attention processing result calculation, the neural network model inference and the image processing, and the performance of the vehicle-mounted system is improved.

[0082] Please refer to Figure 3 , the multiple offset point information corresponding to the reference point includes coordinate information of multiple offset points and interpolation pixel values corresponding to the multiple offset points, and the determination module 130 includes a first determination sub-module 131 and a second determination sub-module 132;

[0083] The first determining submodule 131 is configured to determine a target weight corresponding to the offset point according to the attention weight forwarded by the acquisition module 120 and the coordinate information of the offset point.

[0084] The second determining submodule 132 is configured to determine the attention processing result according to the target weight sent by the first determining submodule 131 and the interpolation pixel value forwarded by the acquisition module 120.

[0085] Specifically, in the embodiments of the present application, the determining module 130 includes the first determining submodule 131 and the second determining submodule 132. In addition, the offset point information acquired by the acquisition module 120 from the storage module 110 includes the coordinate information of the offset point of the feature map and the pixel value (i.e., the interpolation pixel value) of the interpolation pixel point corresponding to the offset point, and the first determining submodule 131 and the second determining submodule 132 complete the online calculation of the attention processing result through these data.

[0086] It should be noted that in the calculation formula of the deformable attention mechanism, the pixel value of the offset point in the tensor (i.e., the feature map) needs to be determined according to the offset point coordinates, that is, W m 'x(p q +Δp mqk ) in the aforementioned formula.

[0087] It should be further noted that in W m 'x(p q +Δp mqk ), the horizontal and vertical coordinates in the offset point coordinates represented by (p q +Δp mqk ) may be both decimal numbers (or floating-point numbers), such as (3.4, 5.2).

[0088] It can be understood that in the case where the horizontal and vertical coordinates are both decimal numbers, (p q +Δp mqk ) is difficult to point to any pixel point in the feature map. Therefore, the pixel of the pixel point corresponding to (p q +Δp mqk ) can be equivalently replaced by the pixel of the pixel pointed to by the coordinates with integer horizontal and vertical coordinates around (p q +Δp mqk ), that is, the pixel value of the interpolation pixel point corresponding to the offset point is used to determine the pixel value of the offset point.

[0089] Therefore, in some embodiments of the present application, W m 'x(p q +Δp mqk ) in the calculation formula of the deformable attention mechanism can be expanded as follows, that is:

[0090] W m ′x(p q +Δp mqk )=(P)=(Q 11 (2-x)(2-)+(Q) 12 (2-)(y-1)

[0091] +(Q 21 (-x1)(2-)+(Q) 22 (-x1)(-y1)

[0092] For a clearer illustration of the implementation methods of this application, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram illustrating application scenarios in some embodiments of this application. That is, as shown... Figure 4 As shown in the above formula, P(x,y) represents the coordinates of the offset point, f() represents the pixel value corresponding to the coordinates within the parentheses, and Q... 11 (x1,y1),Q 12 (x1,y2),Q 21 (y2, y1) and Q 22 (x2, y2) represents the interpolated pixel point corresponding to the offset point.

[0093] Among them, Q 11 Q 12 Q 21 and Q 22 It can be determined by P. For example, if P is (4.4, 4.8), then Q... 11 It can be (4,4), Q 12 It can be (4,5), Q 21 It can be (5,4), Q 22 It can be (5,5).

[0094] Furthermore, Q 11 Q 12 Q 21 and Q 22 The value of P can be determined by rounding up and down. For example, if P is (4.4, 4.8), the rounded-up result of the x-coordinate (4.4) is 5, and the rounded-down result is 4. The rounded-up result of the y-coordinate (4.8) is 5, and the rounded-down result is 4. Therefore, Q corresponding to P can be determined using the x-coordinate values ​​within the range of 4 and 5. 11 Q 12 Q 21 and Q 22 .

[0095] Based on the above background, the embodiments of this application can complete the online calculation of attention processing results by using the coordinate information of the offset points of the determined feature map and the pixel values ​​of the interpolated pixels corresponding to the offset points.

[0096] Specifically, please refer to [the relevant document] again. Figure 3 and Figure 4 Let the offset point be P(x,y) and the interpolated pixel point be Q. 11 (x1,y1),Q 12 (x1,y2),Q 21 (y2, y1) and Q 22 (x2, y2), and let the attention weight be A k Then we have:

[0097] The acquisition module 120 sends a read request to the storage module 110. Based on the received read request, the storage module 110 sends the attention weight A to the acquisition module 120. k The coordinates of the offset point P and the interpolated pixel Q 11 Q 12 Q 21 and Q 22 The pixel value (i.e., the interpolated pixel value), f(Q) 11 f(Q) 12 f(Q) 21 ) and f(Q 22 ).

[0098] After receiving the data sent by the storage module 110, the acquisition module 120 sends the attention weight A to the first determination submodule 131. k The coordinate information of the offset point P, and the interpolated pixel value f(Q) sent to the second determining submodule 132. 11 f(Q) 12 f(Q) 21 ) and f(Q 22 ).

[0099] The first determining submodule 131 determines the coordinate information of the received offset point P(x,y) and Q. 11 (x1,y1),Q 12 (x1,y2),Q 21 (y2, y1) and Q 22 (x2, y2), determine Q 11 Q 12 Q 21 and Q 22 The interpolation pixel coefficients C. Where, Q 11 interpolation pixel coefficient C 11 For (x²-x)(y²-y), Q 12 interpolation pixel coefficient C12 (x2-x)(y-y1), Q 21 the interpolation pixel coefficient C of the interpolation pixel point Q 21 (x-x1)(y2-y), Q 22 the interpolation pixel coefficient C of the interpolation pixel point Q 22 (x-x1)(y-y1).

[0100] Then, the first determining submodule 131 calculates, for each interpolation pixel point Q, the attention weight A k and the product of the interpolation pixel coefficient C of the interpolation pixel point, to obtain the target weight wgt corresponding to the interpolation pixel point Q. The calculation formula of the target weight wgt can be as follows, that is:

[0101] wgt1= A k ·(x2-x)(2-y)

[0102] wgt2= A k ·(x2-x)(y-1)

[0103] wgt3= A k ·(x-x1)(2-y)

[0104] wgt4= A k ·(x-x1)(y-1)

[0105] Then, the first determining submodule 131 sends the target weight wgt corresponding to each interpolation pixel point Q to the second determining submodule 132. The second determining submodule 132 performs inner product operation and summation on wgt and f(Q) corresponding to each interpolation pixel point Q according to the received wgt and f(Q) corresponding to each interpolation pixel point Q, to obtain the operation result res corresponding to P, and the specific process can be referred to the following formula, that is:

[0106] res = f(Q 11 )·wgt1 + (Q 12 )·wgt2 + (Q 21 )·wgt3 + (Q 22 )·wgt4

[0107] Finally, the attention processing result of the feature map can be determined according to the operation result res corresponding to P.

[0108] In this way, in the embodiments of the present application, the target weight can be determined based on the attention weight, the coordinate information of the offset point of the feature map and the interpolation pixel value corresponding to the offset point, and the attention processing result can be determined according to the target weight and the pixel value of the interpolation pixel point, so as to realize online calculation of the attention processing result.

[0109] In some embodiments of the present application, the coordinate information of the offset point includes a decimal part of the coordinate of the offset point, and the first determining sub-module 131 is configured to:

[0110] According to the decimal part of the offset point, determine the interpolation pixel coefficient corresponding to the offset point;

[0111] According to the attention weight and the interpolation pixel coefficient corresponding to the offset point, determine the target weight corresponding to the offset point.

[0112] Specifically, in order to improve the determination efficiency of the interpolation pixel coefficient and reduce the calculation time required by the attention processing result, the coordinate information of the offset point in the embodiments of the present application can include a decimal part of the horizontal and vertical coordinates of the offset point.

[0113] For a clearer description of the embodiments of the present application, please refer again to Figure 4 . Specifically, let P(x, y) represent the offset point, and let Q 11 (x1, y1), Q 12 (x1, y2), Q 21 (y2, y1) and Q 22 (x2, y2) represent the interpolation pixel points corresponding to the offset point, and refer to the expansion of W m 'x(p q +Δp mqk ) in the calculation formula of the deformable attention mechanism mentioned above, that is:

[0114] W m ′x(p q +Δp mqk )=(P)=(Q 11 )(2-x)(2-)+(Q 12 )(2-)(y-1)

[0115] +(Q 21 )(-x1)(2-)+(Q 22 )(-x1)(-y1)

[0116] It can be understood that the coordinates of the interpolation pixel point Q corresponding to the offset point P can be determined by the decimal part of the horizontal and vertical coordinates of the offset point P and the difference of the decimal part with respect to 1. For example, let the decimal part of the horizontal coordinate of the offset point P be x d , and let the decimal part of the vertical coordinate of the offset point P be y d , then the interpolation pixel coefficient C1 of Q 11 is ((1-x d )*(1-y d )), the interpolation pixel coefficient C2 of Q 12 is ((1-x d )*y d ), and the interpolation pixel coefficient C3 of Q21 The interpolation pixel coefficient C3 of the offset point P(x, y) is (x d *(1-y d )) and Q 22 The interpolation pixel coefficient C4 of the offset point P(x, y) is (x d *y d ).

[0117] Based on this, the embodiments of the present application can obtain and use the decimal part of the coordinates of the offset point to complete the determination of the interpolation pixel coefficient. Specifically, in some embodiments as shown in FIG. 1, after the obtaining module 120 sends a read request to the storage module 110, the storage module 110 determines the integer part and the decimal part of the offset point P(x, y) according to the received read request, and sends the attention weight A k , f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ) and the decimal part of the offset point P(x, y) to the obtaining module 120. d and y d . Figure 3

[0118] Wherein, after the obtaining module 120 sends a read request to the storage module 110, the storage module 110 determines the integer part and the decimal part of the offset point P(x, y) according to the received read request, and determines the pixel values of Q 11 , Q 12 , Q 21 and Q 22 , i.e. f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ) according to the integer part of the offset point P(x, y) (or according to the integer part and the decimal part of the offset point P).

[0119] Further, in the case that the storage module 110 determines f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ) and the attention weight A k according to the read request, the storage module 110 can send the attention weight A k , f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ) and the decimal part of the offset point P(x, y) to the obtaining module 120.

[0120] ​Therefore, in the embodiments of the present application, the determination of the interpolation pixel coefficient can be based on the decimal part of the offset point coordinates, so as to reduce the overhead caused by data interaction in the calculation process of the attention processing result to a certain extent, and to efficiently determine the interpolation pixel coefficient.

[0121] In some embodiments of the present application, the second determination sub-module 132 is configured to:

[0122] determine the sub-attention processing result corresponding to each offset point according to the target weight and the interpolation pixel value corresponding to each offset point;

[0123] determine the attention processing result according to the sub-attention processing result corresponding to each offset point.

[0124] It can be understood that in the embodiments of the present application, one feature map can have multiple offset points, for example, in the calculation formula of the deformable attention, one attention head corresponds to K offset points.

[0125] Therefore, for multiple offset points of a feature map, the second determination sub-module 132 (or the determination module 130) of the embodiments of the present application can store the sub-attention processing result corresponding to each offset point in a certain data storage unit (such as a register) of itself after calculating the sub-attention processing result corresponding to each offset point, so as to accumulate the sub-attention processing results of multiple offset points through the data storage unit, thereby obtaining the attention processing result of the feature map.

[0126] Specifically, referring again to Figure 4 and let the sub-attention processing result be the above operation result res, then the second determination sub-module 132 has a register inside, and the second determination sub-module 132 stores the res corresponding to each offset point P in the register after calculating the res corresponding to each offset point P, so as to accumulate the res of each offset point P corresponding to the feature map through the register, thereby obtaining the attention processing result DeformAttn of the feature map, and sending the DeformAttn to the storage module 110 for storage.

[0127] Therefore, in the embodiments of the present application, the attention processing result of the feature map can be determined according to the sub-attention processing result of each offset point in the feature map.

[0128] In some embodiments of the present application, the offset point information corresponding to the reference point includes coordinate information of multiple offset points and interpolation pixel values corresponding to multiple offset points, and the determination module 130 includes a first determination sub-module 131 and a second determination sub-module 132.

[0129] The first determining sub-module 131 is configured to determine an interpolation pixel coefficient corresponding to each offset point according to the coordinate information of the offset points forwarded by the acquisition module 120, construct a coefficient matrix according to the interpolation pixel coefficients corresponding to each offset point, and determine a weight matrix according to the coefficient matrix and the attention weight.

[0130] The second determining sub-module 132 is configured to determine an attention processing result according to the weight matrix sent by the first determining sub-module 131 and the interpolation pixel values corresponding to the plurality of offset points forwarded by the acquisition module 120.

[0131] Specifically, in the inference process of the neural network model, usually involves the processing of multiple channels or multiple images. For example, in one example, an image with a size of 224x224 and a color mode of RGB (Red-Green-Blue) three channels is input into the neural network model to determine the belonging category of the object in the RGB image in the known 1000 categories. The change process of the image can include: (224, 224, 3)-(224, 224, 64)-(112, 112, 128)-(56, 56, 256)-(28, 28, 512)-(14, 14, 512)-(7, 7, 512)-(1, 1, 4096)-(1, 1, 1000).

[0132] It can be understood that in the above change process of the image, the first two parameters in the parentheses represent the size of the image, and the last parameter in the parentheses represents the number of channels of the image.

[0133] Similar to the above change process of the image, when the neural network model includes an attention mechanism, the image of each channel can calculate a corresponding attention processing result, and the image of each channel can be processed through the attention processing result of each channel.

[0134] It can also be understood that a complete image can be understood as being composed of a plurality of sub-images passing through, and therefore the feature map of the embodiment of the application can be composed of a plurality of feature sub-maps. In addition, there can be a plurality of offset points in an image.

[0135] Based on the above background, in the embodiment of the application, in order to realize efficient processing of the multi-channel and multi-offset point of the feature map, or to realize efficient processing of the single-channel and multi-offset point of the feature map, the embodiment of the application can integrate the calculation data corresponding to a plurality of offset points into a matrix, so as to realize parallel calculation of the attention processing results corresponding to the plurality of offset points through matrix operation.

[0136] Specifically, in the embodiments of the present application, the data required for calculating the attention processing result DeformAttn of any feature map can include the target weight wgt and the interpolation pixel value f(Q) of the interpolation pixel point Q corresponding to the offset point P. Thus, for any offset point P, the first determination submodule 131 can determine the interpolation pixel coefficient C of each interpolation pixel point Q corresponding to the offset point P by obtaining the horizontal and vertical coordinates x and y of the offset point P(x, y) forwarded by the acquisition module 120.

[0137] Then, the first determination submodule 131 constructs the coefficient matrix M C corresponding to all offset points P by using the interpolation pixel coefficient C of each interpolation pixel point Q corresponding to each offset point P.

[0138] Then, each element / interpolation pixel coefficient C in the coefficient matrix M C is multiplied by the attention weight A k , so that the coefficient matrix M C composed of the interpolation pixel coefficients C is converted into the weight matrix M wgt composed of the weights wgt. Finally, the first determination submodule 131 sends the weight matrix M wgt to the second determination submodule 132.

[0139] After receiving the weight matrix M wgt sent by the first determination submodule 131 and obtaining the interpolation pixel value f(Q) of each interpolation pixel point Q corresponding to each offset point P forwarded by the acquisition module 120, the second determination submodule 132 constructs the matrix MF(Q) based on the pixel value f(Q) of each interpolation pixel point Q corresponding to each offset point P, and then performs matrix multiplication of the weight matrix M wgt and the matrix MF(Q) to determine the inner product wgt-f(Q) of the target weight wgt and the interpolation pixel value f(Q) of the interpolation pixel point Q, and constructs the matrix M T1 based on the inner product wgt-f(Q) of each interpolation pixel point Q corresponding to each offset point P.

[0140] Finally, the attention processing result of the feature map can be determined based on the matrix M T1 .

[0141] It can be understood that the process of determining the attention processing result DeformAttn of each feature subgraph of the feature map based on the matrix M T1 may include: in the matrix M T1In the case that the inner product (wgt-f(Q)) of each interpolation pixel point Q corresponding to each offset point P in the feature subgraph is included, the vehicle can accumulate each (wgt-f(Q)) corresponding to each offset point P in the feature subgraph, so as to obtain the attention processing result DeformAttn of the subgraph image.

[0142] Thus, in the embodiments of the present application, the interpolation pixel coefficient of the interpolation pixel point corresponding to the offset point can be determined through the coordinate information of the offset point, the coefficient matrix can be constructed according to the interpolation pixel coefficient of the interpolation pixel point corresponding to each offset point, and the determination of the attention processing result can be completed in combination with the matrix operation mode according to the coefficient matrix, the attention weight and the interpolation pixel value, so that the attention processing result can be determined efficiently.

[0143] In addition, it can also be understood that in the embodiments of the present application, the vehicle can also realize efficient processing of the multi-channel multi-offset point of the feature map through other calculation sequences.

[0144] For example, in some embodiments, the offset point information includes the coordinate information of a plurality of offset points P of the feature map and the pixel value f(Q) of the interpolation pixel point Q corresponding to each offset point P, and the vehicle can utilize the interpolation pixel coefficient C of each interpolation pixel point Q corresponding to each offset point P to construct the coefficient matrix M of all offset points P according to the coordinate information of each offset point P. C .

[0145] At the same time, by the pixel value f(Q) of each interpolation pixel point Q corresponding to each offset point P, the matrix M constructed by f(Q) is constructed. f(Q) .

[0146] Then, each element f(Q) in the pixel value matrix M is multiplied by the attention weight A to make each element in the pixel value matrix M converted into the product S of f(Q) and A, and further make M converted into M. f(Q) . k . f(Q) . k . f(Q) . S .

[0147] Finally, the matrix multiplication of M and the coefficient matrix M is calculated, and the above M is obtained. S . C . T .

[0148] For example, in some embodiments, the offset point information includes coordinate information of a plurality of offset points P of the feature map, and pixel values f(Q) of interpolation pixel points Q corresponding to each offset point P. The vehicle can construct a coefficient matrix M corresponding to all offset points P by using interpolation pixel coefficients C of each interpolation pixel point Q corresponding to each offset point P, with respect to the coordinate information of each offset point P. C .

[0149] Then, each interpolation pixel coefficient C in the coefficient matrix M C is multiplied by the pixel value f(Q) of each interpolation pixel point Q corresponding to each offset point P to obtain a product result C*f(Q) of each interpolation pixel point Q corresponding to each offset point P.

[0150] At the same time, the product results C*f(Q) of each interpolation pixel point Q corresponding to each offset point P are summed to obtain the pixel value f(P) of each offset point P, so that the coefficient matrix M C is converted into a pixel value matrix M f(P) composed of the pixel values f(P) of each offset point P.

[0151] Finally, each element in the pixel value matrix M f(P) is multiplied by the attention weight A k , so that the pixel value matrix M f(P) is converted into M T2 , so as to determine the attention processing result of the feature map by M T2 .

[0152] Further, in some embodiments, the product results C*f(Q) of each interpolation pixel point Q corresponding to each offset point P can not be summed, that is, after obtaining the product results C*f(Q) of each interpolation pixel point Q corresponding to each offset point P, so that each element in the coefficient matrix M C is converted into C*f(Q), and thus M C is converted into M tmp , so that each element in M tmp is multiplied by the attention weight A k , so that the matrix M tmp is converted into the matrix M T3 , so that the attention processing result of the feature map can be determined by the matrix M T3 .

[0153] In some embodiments of the present application, the offset point information includes offset point pixel values of a plurality of offset points, and the determination module 130 is configured to:

[0154] determine a sub-attention processing result corresponding to the offset point according to the offset point pixel value and the attention weight forwarded by the acquisition module 120; ​​

[0155] According to the sub-attention processing results corresponding to the plurality of offset points, the attention processing result is determined.

[0156] Specifically, in the embodiments of the present application, the storage module 110 stores the pixel interpolation of each offset point, that is, the pixel value of each offset point, which is calculated in advance. Further, the acquisition module 120 can read the pixel value of each offset point and the attention weight, and send the pixel value and the attention weight to the determination module 130, so that the determination module 130 outputs the attention processing result.

[0157] Specifically, please refer to Figure 2 again. The offset point information stored by the storage module 110 can include the pixel value of the offset point and the attention weight. Among them, the pixel value of the offset point can be understood as the W m x(p q +Δp mqk ) term in the above deformable attention calculation formula, and the attention weight can be understood as the A mqk term in the above deformable attention calculation formula.

[0158] Optionally, in some embodiments of the present application, the storage module 110 can store the coordinate information of the offset point, the pixel value of the interpolation pixel point corresponding to the offset point, and the attention weight inside. In addition, the storage module 110 can send the coordinate information of the offset point and the pixel value of the interpolation pixel point corresponding to the offset point to the pre-set calculation unit such as CPU, so that the calculation unit determines the interpolation pixel coefficient according to the coordinate information, and in the case of determining the pixel value of the offset point according to the interpolation pixel coefficient and the pixel value of the interpolation pixel point, writes the pixel value of the offset point to the storage module 110.

[0159] Therefore, the storage module 110 can send the attention weight and the pixel value of the offset point to the acquisition module 120 according to the read request sent by the acquisition module 120, and the acquisition module 120 can forward the attention weight and the pixel value of the offset point to the determination module 130 in the case of receiving the attention weight and the pixel value of the offset point, so that the determination module 130 determines the attention processing result of the feature map according to the attention weight and the pixel value of the offset point.

[0160] In this way, in the embodiments of the present application, the attention processing result of the feature map can be determined according to the attention weight of the feature map and the pixel value of the offset point in the feature map, so that the attention processing result of the feature map is determined efficiently.

[0161] Please refer to Figure 5In some embodiments of the present application, the offset point information corresponding to the reference point includes coordinate information of the offset points and interpolation pixel values corresponding to the offset points, and the determining module 130 includes a first determining sub-module 131 and a second determining sub-module 132.

[0162] The first determining sub-module 131 is configured to determine the interpolation pixel coefficients corresponding to the offset points according to the coordinate information of the offset points forwarded by the obtaining module 120, and determine the offset point pixel values of the offset points according to the interpolation pixel coefficients corresponding to the offset points and the interpolation pixel values corresponding to the offset points forwarded by the obtaining module 120.

[0163] The second determining sub-module 132 is configured to determine the attention processing result according to the attention weights forwarded by the obtaining module 120 and the offset point pixel values sent by the first determining sub-module 131.

[0164] Specifically, in the embodiments of the present application, the offset point information stored by the storage module 110 includes coordinate information of the offset points P of the feature map and pixel values f(Q) of the interpolation pixel points Q corresponding to each offset point P. Wherein, the interpolation pixel points Q include Q 11 (x1,y1), Q 12 (x1,y2), Q 21 (y2,y1) and Q 22 (x2,y2), so that the pixel values f(Q) of the interpolation pixel points Q include f(Q 11 ), f(Q 12 ), f(Q 21 ) and f(Q 22 ).

[0165] The obtaining module 120 sends a read request to the storage module 110. The storage module 110 sends the attention weights A k , the coordinate information of the offset points P and f(Q 11 ), f(Q 12 ), f(Q 21 ) and f(Q 22 ) to the obtaining module 120 according to the received read request.

[0166] After receiving each data sent by the storage module 110, the obtaining module 120 sends the coordinate information of the offset points P and the pixel values f(Q 11 ), f(Q 12 ), f(Q 21 ) and f(Q 22 ) to the first determining sub-module 131, and sends the attention weights A k to the second determining sub-module 132.

[0167] The first determining sub-module 131 determines the coordinate information of the interpolation pixel point Q 11 (x1,y1), Q 12 (x1,y2), Q 21 (y2,y1), and Q 22 (x2,y2), determines the interpolation pixel coefficient C of Q 11 , Q 12 , Q 21 , and Q 22 . Wherein, the interpolation pixel coefficient C of Q 11 is (x2-x)(y2-y), the interpolation pixel coefficient C of Q 11 is (x2-x)(y-y1), the interpolation pixel coefficient C of Q 12 is (x2-x1)(y2-y), and the interpolation pixel coefficient C of Q 12 is (x-x1)(y2-y). 21 21 22 22

[0168] For each interpolation pixel point Q, the first determining sub-module 131 calculates the product of the pixel value f(Q) of the interpolation pixel point and the interpolation pixel coefficient C, and calculates the sum value of the product of the pixel value f(Q) and the interpolation pixel coefficient C of each interpolation pixel point corresponding to the offset point, thereby obtaining the pixel value f(P) of the offset point, i.e., the offset point pixel.

[0169] Subsequently, the first determining sub-module 131 sends the pixel value f(P) of the offset point to the second determining sub-module 132. The second determining sub-module 132 calculates the product of the attention weight A k and the pixel value f(P) of the offset point P according to the received attention weight A k and the pixel value f(P) of the offset point, thereby obtaining the operation result res corresponding to the offset point P.

[0170] Finally, the second determining sub-module 132 can determine the attention processing result of the feature map according to the operation result res corresponding to P.

[0171] Thus, in the embodiments of the present application, the attention weight, the coordinate information of the offset point of the feature map, and the pixel value of the interpolation pixel point corresponding to the offset point can be obtained to determine the offset point pixel value, and the attention processing result can be determined according to the offset point pixel value and the attention weight, thereby realizing the online calculation of the attention processing result.

[0172] ​​​​In addition, it can be understood that the pixel value f(P) of the offset point can be obtained by multiplying the pixel value f(Q) of each interpolation pixel point Q with the corresponding interpolation pixel coefficient C and then accumulating the product results corresponding to each interpolation pixel point Q.

[0173] It can also be understood that the interpolation pixel coefficients C corresponding to the interpolation pixel points Q respectively can reflect the distance relationship between the interpolation pixel points Q and the offset point P. That is, the closer the interpolation pixel points Q and the offset point P, the higher the interpolation pixel coefficient C of the interpolation pixel points Q, and vice versa.

[0174] In addition, it can also be understood that the value of the offset point P is an integer.

[0175] In some embodiments of the present application, the offset point information corresponding to the reference point includes coordinate information of the offset points and interpolation pixel values corresponding to the offset points, and the determination module 130 includes a first determination submodule 131 and a second determination submodule 132.

[0176] The first determination submodule 131 is configured to determine the pixel weighting result corresponding to the offset point according to the interpolation pixel value corresponding to the offset point and the attention weight forwarded by the acquisition module 120.

[0177] The second determination submodule 132 is configured to determine the attention processing result according to the pixel weighting result corresponding to the offset point sent by the first determination submodule 131 and the interpolation pixel coefficient determined by the coordinate information forwarded by the acquisition module 120.

[0178] Specifically, please refer to Figure 6 , Figure 6 is a schematic diagram of the image frame processing chip 100 in some embodiments of the present application. That is, the offset point information stored in the storage module 110 includes the coordinate information of the offset points P of the feature map and the pixel value f(Q) of the interpolation pixel point Q corresponding to each offset point P. Wherein, the interpolation pixel point Q includes Q 11 (x1,y1), Q 12 (x1,y2), Q 21 (y2,y1) and Q 22 (x2,y2), so the pixel value f(Q) of the interpolation pixel point Q includes f(Q 11 ), f(Q 12 ), f(Q 21 ) and f(Q 22 ).

[0179] The acquisition module 120 sends a read request to the storage module 110. The storage module 110 sends the attention weight A k , the coordinate information of the offset point P and f(Q11 f(Q) 12 f(Q) 21 ) and f(Q 22 ).

[0180] After receiving the data sent by the storage module 110, the acquisition module 120 sends the attention weight A to the first determination submodule 131. k and interpolated pixel value f(Q) 11 f(Q) 12 f(Q) 21 ) and f(Q 22 ), and send the coordinate information of offset point P to the second determination submodule 132.

[0181] The first determining submodule 131 determines the attention weight A based on the received attention weight A. k Pixel value f(Q) 11 f(Q) 12 f(Q) 21 ) and f(Q 22 ), calculate attention weight A k respectively with f(Q) 11 f(Q) 12 f(Q) 21 ) and f(Q 22 The product of ) yields the pixel-weighted result of each interpolated pixel Q, i.e., A. k *f(Q 11 A) k *f(Q 12 A) k *f(Q 21 ) and A k *f(Q 22 ), and send A to the second determination submodule 132 k *f(Q 11 A) k *f(Q 12 A) k *f(Q 21 ) and A k *f(Q 22 ).

[0182] The second determining submodule 132 determines Q based on the coordinate information of the received offset point P. 11 Q 12 Q 21 and Q 22 The interpolation pixel coefficients C. Where, Q 11 interpolation pixel coefficient C 11 For (x²-x)(y²-y), Q 12 interpolation pixel coefficient C 12(x2-x)(y-y1), Q 21 the interpolation pixel coefficient C of Q 21 (x-x1)(y2-y), Q 22 the interpolation pixel coefficient C of Q 22 (x-x1)(y-y1).

[0183] Further, the second determining submodule 132 performs corresponding multiplication operation and summation on A k *f(Q 11 ), A k *f(Q 12 ), A k *f(Q 21 ) and A k *f(Q 22 ) and the interpolation pixel coefficient C of Q 11 , Q 12 , Q 21 and Q 22 , so as to obtain the operation result res corresponding to the offset point P.

[0184] Finally, the second determining submodule 132 can determine the attention processing result of the feature map according to the operation result res corresponding to P.

[0185] In this way, in the embodiments of the present application, the attention weight, the coordinate information of the offset point of the feature map and the pixel value of the interpolation pixel point corresponding to the offset point can be obtained, the pixel weighting result can be determined, and the attention processing result of the feature map can be determined according to the pixel weighting result and the target coefficient determined through the coordinate information of the offset point, so as to realize online calculation of the attention processing result.

[0186] In some embodiments of the present application, the coordinate information of the plurality of offset points corresponding to the reference point is obtained by randomly selecting pixels of the feature map based on the neural network model trained in advance.

[0187] In this way, in the embodiments of the present application, the coordinate information of the plurality of offset points can be determined based on the neural network model and the random pixels.

[0188] In some embodiments of the present application, the first determining submodule 131 comprises a central processing unit (CPU), and the second determining submodule 132 comprises an NPU (Neural Processing Units, embedded neural network processor).

[0189] In some embodiments of the present application, the first determining submodule 131 and the second determining submodule 132 both comprise an NPU (Neural Processing Units, embedded neural network processor).

[0190] The embodiment of the present application also provides a vehicle, which is equipped with the image frame processing chip 100.

[0191] In the description of the present specification, the description referring to the terms "specifically", "further", "particularly", "can be understood", and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not intend to refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0192] Any process or method descriptions in flow charts or described herein in other sequences can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for performing specific logic functions or steps in the process. The scope of preferred embodiments of the present application includes the additional implementation in which the functions can be performed in different order, including substantially simultaneously or in reverse order, and the functions can be performed in accordance with the involved functions, and those skilled in the art should understand.

[0193] Although the embodiments of the present application have been shown and described above, it can be understood that the above-described embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. An image frame processing chip, characterized in that, include: The storage module is configured to store multiple offset point information and attention weights corresponding to reference points of the feature map of the image frame. The multiple offset point information corresponding to the reference point includes the coordinate information of the multiple offset points and the interpolated pixel values ​​corresponding to the multiple offset points. The coordinate information of the multiple offset points corresponding to the reference point is obtained by randomly selecting pixels from the feature map based on a pre-trained neural network model. The acquisition module is configured to send a data read request to the storage module and receive the multiple offset point information and attention weights sent by the storage module. The determining module is configured to determine the attention processing result of the feature map based on the multiple offset point information forwarded by the obtaining module and the attention weight, and to store the attention processing result in the storage module to complete the processing of the image frame. The determining module includes a first determining submodule and a second determining submodule; When the acquisition module is configured to forward the attention weight and the coordinate information of the offset point to the first determining submodule, the first determining submodule is configured to determine the target weight corresponding to the offset point based on the attention weight forwarded by the acquisition module and the coordinate information of the offset point, and the second determining submodule is configured to determine the attention processing result based on the target weight sent by the first determining submodule and the interpolated pixel value forwarded by the acquisition module.

2. The chip according to claim 1, characterized in that, The coordinate information of the offset point includes the decimal part of the offset point coordinates, and the first determining submodule is configured as follows: Based on the fractional part of the offset point, the interpolation pixel coefficient corresponding to the offset point is determined; The target weight corresponding to the offset point is determined based on the attention weight and the interpolation pixel coefficient corresponding to the offset point.

3. The chip according to claim 1, characterized in that, The second determining submodule is configured as follows: Based on the target weight and the interpolated pixel value corresponding to each offset point, determine the sub-attention processing result corresponding to each offset point; The attention processing result is determined based on the sub-attention processing result corresponding to each offset point.

4. The chip according to claim 1, characterized in that, The first determining submodule is configured to determine the interpolation pixel coefficients corresponding to the offset points based on the coordinate information of the offset points forwarded by the obtaining module, construct a coefficient matrix based on the interpolation pixel coefficients corresponding to each offset point, and determine a weight matrix based on the coefficient matrix and the attention weights. The second determining submodule is configured to determine the attention processing result based on the weight matrix sent by the first determining submodule and the interpolated pixel values ​​corresponding to the multiple offset points forwarded by the acquisition module.

5. The chip according to claim 1, characterized in that, The first determining submodule is configured to determine the interpolation pixel coefficient corresponding to the offset point based on the coordinate information of the offset point forwarded by the acquisition module, and to determine the offset point pixel value based on the interpolation pixel coefficient corresponding to the offset point and the interpolation pixel value corresponding to the offset point forwarded by the acquisition module. The second determining submodule is configured to determine the attention processing result based on the attention weight forwarded by the acquisition module and the offset pixel value sent by the first determining submodule.

6. The chip according to claim 1, characterized in that, The first determining submodule is configured to determine the pixel weighting result corresponding to the offset point based on the interpolated pixel value and the attention weight forwarded by the acquisition module; The second determining submodule is configured to determine the attention processing result based on the pixel weighting result corresponding to the offset point sent by the first determining submodule and the interpolation pixel coefficient determined by the coordinate information forwarded by the acquisition module.

7. The chip according to claim 1, characterized in that, The offset point information includes the offset point pixel values ​​of the plurality of offset points, and the determining module is configured as follows: Based on the offset pixel value forwarded by the acquisition module and the attention weight, the sub-attention processing result corresponding to the offset point is determined; The attention processing result is determined based on the sub-attention processing results corresponding to the multiple offset points.

8. A vehicle, characterized in that, Includes the chip described in any one of claims 1-7.

Images