Image fusion method and device, intelligent chip and storage medium

Abstract

The disclosure provides an image fusion method and device, an intelligent chip and a storage medium, relates to the field of artificial intelligence, in particular to the technical field of computer vision and image fusion, and can be applied to scenarios such as automatic driving, intelligent transportation and remote monitoring. The specific implementation scheme of the image fusion method is as follows: determining pixels of at least two images to be fused at each pixel position to obtain at least two pixels; determining a fusion weight for the at least two pixels according to a predetermined weight table, each pixel position and the number of target pixels participating in fusion in the at least two pixels; and fusing the at least two pixels according to the fusion weight to obtain a pixel at each pixel position in a fused image, wherein the predetermined weight table is generated in advance according to the pixel position.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the technical field of computer vision and image fusion, and can be applied to scenarios such as automatic driving, intelligent transportation, and remote monitoring. BACKGROUND

[0002] With the development of computer technology and electronic technology, image processing technology is widely used in many fields. For example, in the remote monitoring of an automatic driving vehicle, an image fusion algorithm is usually used to fuse multiple images to evaluate the running status of the automatic driving vehicle according to the fusion result. SUMMARY

[0003] The present disclosure aims to provide an image fusion method, device, intelligent chip and storage medium that facilitate online real-time image fusion.

[0004] According to one aspect of the present disclosure, an image fusion method is provided, comprising: determining pixels of at least two images to be fused at each pixel position to obtain at least two pixels; determining fusion weights for the at least two pixels according to a predetermined weight table, each pixel position, and the number of target pixels participating in fusion in the at least two pixels; and fusing the at least two pixels according to the fusion weights to obtain a pixel at each pixel position in a fused image, wherein the predetermined weight table is generated in advance according to the pixel position.

[0005] According to another aspect of the present disclosure, an image fusion device is provided, comprising: a pixel determination module configured to determine pixels of at least two images to be fused at each pixel position to obtain at least two pixels; a first weight determination module configured to determine fusion weights for the at least two pixels according to a predetermined weight table, each pixel position, and the number of target pixels participating in fusion in the at least two pixels; and a pixel fusion module configured to fuse the at least two pixels according to the fusion weights to obtain a pixel at each pixel position in a fused image, wherein the predetermined weight table is generated in advance according to the pixel position.

[0006] According to another aspect of the present disclosure, an intelligent chip is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image fusion method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the image fusion method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, there is provided a computer program product comprising computer programs / instructions stored on at least one of a readable storage medium and an electronic device, which, when executed by a processor, implement the image fusion method provided by the present disclosure.

[0009] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0011] Figure 1 is a schematic diagram of an application scenario of an image fusion method, device and intelligent chip according to an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of an image fusion method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of the principle of generating a base weight table according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of the principle of implementing an image fusion method according to an embodiment of the present disclosure;

[0015] Figure 5 is a structural block diagram of an image fusion device according to an embodiment of the present disclosure; and

[0016] Figure 6 is a block diagram of an intelligent chip for implementing an image fusion method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0018] In remote monitoring of an autonomous vehicle, an image fusion algorithm is usually used to fuse images. Generally, the image fusion algorithm is implemented by a main processor CPU. However, limited by limited communication bandwidth and limited computing resources, if the images are fused in real time, the overall performance and stability of the system will be reduced. This is because the CPU needs to occupy a large amount of resources in the process of implementing the image fusion algorithm.

[0019] In the fusion process of multiple images, a serial loop operation is usually used, for example, first fuse any two images in the multiple images to obtain a first fused image. Then fuse the first fused image and any one of the multiple images that has not been fused until the fusion of the multiple images is completed. This method has the problem of low fusion efficiency.

[0020] To solve this problem, the present disclosure provides an image fusion method, device, intelligent chip and storage medium. The following will first combine Figure 1 The application scenario of the image fusion method, device and intelligent chip provided by the present disclosure is described.

[0021] Figure 1 The application scenario of the image fusion method, device and intelligent chip provided by the present disclosure is described.

[0022] As Figure 1 shown, the application scenario 100 of this embodiment can include an autonomous vehicle 110, which can travel on a road. The autonomous vehicle 110 can be integrated with multiple image acquisition devices, for example, to acquire images reflecting the environment of the autonomous vehicle 110, and send the images to an electronic device 120 for remote monitoring via a network.

[0023] In an embodiment, monitoring devices 130 can also be provided on both sides of the road, which can monitor the environmental images of the road and send the monitored images to the electronic device 120 via a network.

[0024] The electronic device 120 can be various electronic devices with processing functions, including but not limited to laptop computers, desktop computers and servers, etc. The electronic device 120 can fuse the received images reflecting the environment of the autonomous vehicle 110 to form a panoramic image, for example, to achieve all-around remote monitoring of the autonomous vehicle 110. In this way, the remote monitoring personnel can determine from the panoramic image whether the autonomous vehicle 110 can operate normally and whether the driving of the autonomous vehicle 110 needs to be remotely controlled.

[0025] In an embodiment, the electronic device 120 can be integrated with a smart chip having an image fusion function to fuse the images received by the electronic device 120. The smart chip can be, for example, a Field Programmable Gate Array (FPGA) or the like, which is not limited in the present disclosure.

[0026] It should be noted that the image fusion method provided by the present disclosure can be executed by the electronic device 120, and specifically can be executed by the smart chip integrated in the electronic device 120. Accordingly, the image fusion apparatus provided by the present disclosure can be arranged in the electronic device 120, and specifically can be integrated in the smart chip integrated in the electronic device.

[0027] It should be understood that Figure 1 The number and type of the autonomous driving vehicle 110, the electronic device 120, and the monitoring device 130 in the system 100 are only illustrative. According to the implementation needs, there can be any number and type of autonomous driving vehicles 110, electronic devices 120, and monitoring devices 130.

[0028] The following will be described in detail Figures 2 to 4 The image fusion method provided by the present disclosure will be described in detail.

[0029] Figure 2 is a flowchart of the image fusion method according to an embodiment of the present disclosure.

[0030] As shown in Figure 2 The image fusion method 200 of the embodiment can include operations S210-S230.

[0031] In operation S210, the pixels of at least two images to be fused at each pixel position are determined to obtain at least two pixels.

[0032] According to an embodiment of the present disclosure, the at least two images to be fused can be, for example, images preprocessed to a fixed size. The fixed size can be, for example, any one of 640x480, 1024x768, 1600x1200, and the like. At each pixel position within the fixed size, the at least two images have one pixel. For example, if the fixed size is 640x480 and the pixel position is (2, 3), the pixel at the 2nd row and 3rd column of each image can be located to obtain one pixel. If the at least two images are N, for the pixel position (2, 3), N pixels can be located to obtain N pixels, and the N pixels belong to the N images to be fused, respectively. N is a natural number greater than or equal to 2.

[0033] In operation S220, the fusion weights for the at least two pixels are determined according to the predetermined weight table, each pixel position, and the number of target pixels participating in the fusion among the at least two pixels.

[0034] According to an embodiment of the present disclosure, the predetermined weight table is generated in advance according to pixel positions. For example, this embodiment can deduce the fusion weights for each pixel position according to the principle of the fusion algorithm, arrange the fusion weights according to the pixel positions to form the predetermined weight table. The value of the fusion weight is related to the pixel position. For example, a weight function can be used to calculate the fusion weight of each pixel position.

[0035] In an embodiment, the expression of the fusion of the at least two images can be determined in advance according to the principle of the serial loop operation. For example, if the principle of the fusion of two images can be represented by the expression F = w1f1 + w2f2, then the expression of the fusion of three images can be represented by F = w1(w1f1 + w2f2) + w2f3. The expression of the fusion of four images can be represented by F = w1[w1(w1f1 + w2f2) + w2f3] + w2f4. Similarly, the expression of the fusion of N images can be deduced. Wherein, w1 and w2 are weight functions related to pixel positions, f1, f2, f3, f4, … represent the pixel values of the first, second, third, fourth, … images in the N images to be fused, and F represents the pixel value of the pixel in the fused image. In this embodiment, the expression of the fusion of the at least two images can be expanded into a polynomial to obtain F = c1f1 + c2f2 + … + cNfN. Wherein, c1, c2, …, cN are the fusion weights of the N images, and are functions of pixel positions. This embodiment can calculate the fusion weights of the pixels in the N images according to all pixel positions of the pixels in the images, and store them in the predetermined storage space in the form of a table to obtain the predetermined weight table. N f N N

[0036] For example, this embodiment can set a predetermined weight table for each value of N. Operation S210 can determine the predetermined weight table set for N with a value of P according to the number P of target pixels. Then, the weight combination corresponding to the pixel position in the predetermined weight table is determined to obtain the fusion weight of the target pixel. It can be understood that the fusion weight of the other pixels in the at least two pixels except the target pixel can be, for example, a second predetermined value. If the number of target pixels is 1, the fusion weight of the target pixel can be determined as 1. The second predetermined value can be, for example, 0 or any value close to 0, which is not limited in the present disclosure. Wherein, P is a natural number less than or equal to N.

[0037] ​​For example, each of the at least two images can have annotation information indicating whether each pixel in the image participates in fusion. The annotation information can be pre-annotated by a human or annotated according to any rule, which is not limited in the present disclosure. In this way, the embodiment can determine the target pixel participating in fusion from the at least two pixels according to the annotation information.

[0038] In operation S230, the at least two pixels are fused according to the fusion weight to obtain a pixel at each pixel position in the fused image.

[0039] The embodiment can weight the pixel values of the at least two pixels according to the fusion weight to obtain a pixel value of a pixel at the pixel position in the fused image. The embodiment can obtain the pixel values of the pixels at all pixel positions in the fused image by fusing the pixels at all pixel positions of the image of a fixed size according to the fusion weight, thereby obtaining the fused image.

[0040] The technical solution of the embodiment of the present disclosure can improve the efficiency of image fusion and reduce the amount of calculation in the image fusion process by determining the fusion weight of each pixel by means of the predetermined weight table generated in advance according to the pixel position when fusing the image. In this way, the occupation of communication bandwidth and computing resources can be reduced when fusing the image in real time online, which is conducive to improving the stability and overall performance of the system.

[0041] In an embodiment, the predetermined weight table can store only the base weight, for example. For example, only the values of the denominators of the polynomials F=c1f1+c2f2+…+c N f N c1, c2, …, c N in the above description can be stored as base weights in the predetermined weight table. In this way, the complexity of the predetermined weight table can be reduced, and the efficiency of reading data from the predetermined weight table can be improved. Accordingly, in the embodiment, the base weight for each pixel position in the predetermined weight table can be obtained when determining the fusion weight. Subsequently, the fusion weight for the at least two pixels can be determined according to the base weight and the number of target pixels.

[0042] For example, if the denominators of w1 and w2 are both (a+b), the embodiment can determine the base weight according to the number of images that need to be fused in the actual scene. For example, if the number of images that need to be fused in the image fusion scene is usually not greater than M, the base weights stored in the predetermined weight table can include (a+b), (a+b) 2 , (a+b) 3 , …, (a+b) (M-1) , where M is a natural number greater than or equal to N.

[0043] In an embodiment, all the base weights for each pixel position in the predetermined weight table can be obtained, i.e., (a+b), (a+b) 2 , (a+b) 3 , …, (a+b) (M-1) are obtained. Alternatively, the required base weights can be obtained from all the base weights for each pixel position stored in the predetermined weight table according to the number P of target pixels. For example, the base weights (a+b), (a+b) 2 , …, (a+b) (P-1) are obtained. It can be understood that the base weights for each pixel position in the predetermined weight table can be sorted according to the indices of (a+b) from small to large, and this embodiment can obtain the base weights ranked in the first (P-1) positions among all the base weights for each pixel position, thereby obtaining the values of (a+b), (a+b) 2 , …, (a+b) (P-1 .

[0044] In an embodiment, after obtaining all the base weights, the embodiment can first determine the target base weights among the obtained base weights according to the number of target pixels. If the number of target pixels is P, the determined target base weights can be the values of (a+b), (a+b) 2 , …, (a+b) (P-1 , for example, can be the base weights ranked in the first (P-1) positions among all the obtained base weights, and the disclosure does not limit this.

[0045] On the basis of obtaining the base weights, the embodiment can determine the values of the numerators of the fusion weights c1, c2, …, c N , for example, according to the number of target pixels. Subsequently, the ratios of the determined values of the numerators to the corresponding denominators are calculated, thereby obtaining the fusion weights c1, c2, …, c N .

[0046] It can be understood that if the number of target pixels is 1, the base weights do not need to be obtained from the predetermined weight table, and the pixel value of the target pixel can be directly taken as the pixel value of the pixel at the corresponding pixel position of the fusion image. Accordingly, the operation S220 described above can be performed in response to the number of target pixels being greater than 1.

[0047] Figure 3 is a schematic diagram of the principle of generating a base weight table according to an embodiment of the disclosure.

[0048] According to the embodiment of the present disclosure, a fixed value can be set for the pixels in the predetermined image region that need to be fused according to actual needs, and the fixed value is taken as the value of the weight function. For example, if a certain pixel position is located in the predetermined image region, the values of w1 and w2 in the weight function used when determining the fusion weight of the pixel position can be fixed values. In this way, for the target pixels that need to be fused, the fusion weight is a first predetermined value associated with the number of target pixels. For example, if w1 and w2 are both fixed values, and the number of target pixels is P, according to the polynomial F = c1f1 + c2f2 + … + cPfP, the fusion weights of the P target pixels are w1, w1*w2, w1*w2, w1*w2, w1*w2, …, w1*w2, w2, respectively. N f N (p-1) （p-2) (p-3) (p-4)

[0049] For example, if the pixel position is located outside the predetermined image region, the base weight can be obtained by querying the predetermined weight table, and the fusion weight of at least two pixels can be determined according to the base weight and the number P of target pixels. It can be understood that the region outside the predetermined image region may, for example, include a plurality of regions located at a plurality of predetermined directions of the predetermined image region. The plurality of predetermined directions may, for example, be left direction, right direction, up direction, down direction, etc. Specifically, the embodiment can obtain the base weight of each pixel position in the predetermined weight table when each pixel position is located at any of the plurality of predetermined directions of the predetermined image region. When each pixel position is located at other positions (for example, located in the predetermined image region) outside the plurality of predetermined directions of the predetermined image region, the fusion weight of the target pixels can be determined as the first predetermined value associated with the number of target pixels described above.

[0050] The embodiment can make the fusion effect of at least two images more in line with actual needs by setting a predetermined image region and determining the fusion weight by querying the predetermined weight table only for the pixel positions located at any of the plurality of predetermined directions of the predetermined image region. For example, the predetermined image region can be the central region of at least two images, because the importance of the pixels in the central region of different images to the fused image is usually equal, and a fixed fusion weight can be set for the pixels in the central region. The pixels in the edge region are usually less important as the distance from the center increases, and the base weight associated with the pixel position can be set for the pixels in the edge region. In this way, the technical solution of the embodiment can improve the accuracy and expressiveness of the fused image obtained by fusion.

[0051] ​​​​​In one embodiment, different values ​​can be set for the parameters in the weighting function for pixels at different orientations within a predetermined image region, so that the determined fusion weights better meet actual needs and better reflect the distance between the pixel position and the predetermined image region. Accordingly, in this embodiment, the predetermined weight table may include multiple sub-tables for multiple predetermined orientations. When obtaining the basic weights, this embodiment can first determine the orientation of each pixel position relative to the predetermined image region, for example, any orientation. Subsequently, this embodiment can determine the sub-table corresponding to that any orientation as the target sub-table and obtain the basic weights for each pixel position from the target sub-table. For example, for any orientation, the parameters in the weighting function can be determined based on the position of the boundary line of the predetermined image region at that orientation. Since the position of the boundary line of the predetermined image region is different for different orientations, the calculation formula for the denominator differs when determining the basic weights, and correspondingly, the calculated basic weights also differ. In this embodiment, the values ​​of the denominator for the weights at different pixel positions calculated using the same formula can be stored as a sub-table.

[0052] In one embodiment, fusion weights related to pixel position can be set only for pixels within a predetermined range at the four vertices of the image, while for other areas, a weight function with fixed values ​​for w1 and w2 is used. This reduces the computational cost of image fusion while ensuring a smoother and more harmonious fused image. This is because images within the predetermined range at the four vertices typically exhibit significant distortion; by using a weight function related to pixel position for interpolation fusion, the impact of image distortion on image fusion can be mitigated, thereby improving the smoothness and harmony of the fused image.

[0053] For example, such as Figure 3 As shown in embodiment 300, when generating the predetermined weight table, it can first be determined whether the row containing each pixel position is outside the row containing the predetermined image region 310 (for example, it can be a direction parallel to the X-axis of the OXY coordinate system constructed based on the image), and whether the column containing each pixel position is outside the column containing the predetermined image region 310 (for example, it can be a direction parallel to the Y-axis of the OXY coordinate system constructed based on the image). If so, the denominator of the fusion weight is calculated according to the weight function related to the pixel position, and the calculated denominator values ​​are arranged sequentially according to the pixel positions to construct the predetermined weight table. If the row containing each pixel position is inside the row containing the predetermined image region 310, or the column containing each pixel position is inside the column containing the predetermined image region 310, the fusion weight can be determined to be the first predetermined value associated with the number of target pixels described above, and it does not need to be stored in the predetermined weight table.

[0054] Accordingly, when obtaining the basic weights in the predetermined weight table, it can first be determined whether the row where each pixel position is located is outside the row where the predetermined image region is located, and whether the column where each pixel position is located is outside the column where the predetermined image region is located. If so, it is determined that each pixel position is located in any of the multiple predetermined orientations of the predetermined image region, and the predetermined weight table is queried to obtain the basic weights.

[0055] In one embodiment, such as Figure 3 As shown, the predetermined image region 310 can be a region enclosed by two boundary rows 311 and 312 and two boundary columns 313 and 314. That is, the predetermined image region 310 can be a rectangular region. Correspondingly, multiple predetermined orientations may include, for example, the orientations of four target regions 320, 330, 340, and 350 enclosed by the two boundary rows 311 and 312, the two boundary columns 313 and 314, and the image boundary. It is understood that the rows containing the four target regions 320, 330, 340, and 350 are all located outside the rows containing the predetermined image region 310, and the columns containing the four target regions 320, 330, 340, and 350 are all located outside the columns containing the predetermined image region 310.

[0056] In one embodiment, an interpolation algorithm can be used to determine the fusion weights, for example. When generating a predetermined weight table, for the four target regions 320, 330, 340, and 350, the denominator of the weights in the weight function can be determined by the vertical distance between any pixel location in the four target regions and the two nearest boundaries of the predetermined image region 310. For example, for a pixel location in target region 320, the denominator of the fusion weight for that pixel location can be determined based on the vertical distance between that pixel location and boundary row 311 and the vertical distance between that pixel location and boundary column 313. For a pixel location in target region 330, the denominator of the fusion weight for that pixel location can be determined based on the vertical distance between that pixel location and boundary row 311 and the vertical distance between that pixel location and boundary column 314. For a pixel location in target region 340, the denominator of the fusion weight for that pixel location can be determined based on the vertical distance between that pixel location and boundary row 312 and the vertical distance between that pixel location and boundary column 313. For a pixel location in the target region 350, the denominator value of the fusion weight for that pixel location can be determined based on the vertical distance between the pixel location and the boundary row 312 and the vertical distance between the pixel location and the boundary column 314.

[0057] For example, let's define the Y-axis coordinates of boundary row 311 as `top_window`, boundary row 312 as `bottom_window`, boundary column 313 as `left_window`, boundary column 314 as `right_window`, the Y-axis coordinates of each pixel's row as `rows`, and the X-axis coordinates of each pixel's column as `cols`. Then, for two images to be merged, the fusion principle of the two pixels participating in the fusion at each pixel location can be determined using the following rules:

[0058] If rows < top_window and cols < left_window, then:

[0059]

[0060] If rows < top_window and cols > right_window, then:

[0061]

[0062] If rows > bottom_window and cols < left_window, then:

[0063]

[0064] If rows > bottom_window and cols > right_window, then:

[0065]

[0066] Accordingly, when generating the predetermined weight table, the orientation of the pixel position relative to the predetermined image region can be determined first. When the pixel position is located in the target region 320, the basic weight can be determined as 1 / (top_window-rows+left_window-cols). For example, if (top_window-rows+left_window-cols) is set... window If (-rows) = a and (left_window-cols) = b, then the basic weight can include the reciprocals of (a+b) raised to the power of 1, 2, ..., (M-1), for example, it can include 1 / (a+b) or 1 / (a+b). 2 1 / (a+b) 3 ..., 1 / (a+b) (M-1)Thus, for all pixel positions located in the target region 320, a sub-table can be constructed. Similarly, for all pixel positions located in the target region 330, all pixel positions located in the target region 340, and all pixel positions located in the target region 350, a sub-table can be constructed respectively, and the four sub-tables can constitute the basic weight table. In this embodiment, the basic weight is represented by the reciprocal of the denominator, which can avoid the division operation occupying large computing resources in the process of real-time image fusion, improve the image fusion efficiency, and facilitate the implementation of real-time image fusion technology.

[0067] According to an embodiment of the present disclosure, after obtaining the basic weight corresponding to each pixel position from the predetermined weight table, the target basic weight of the target pixel in the basic weight can be determined according to the number of target pixels, for example, by using the method described above. Subsequently, the basic weight of each target pixel in the target basic weight can be determined according to the polynomial interpolation principle and the arrangement order of the target pixel in at least two pixels (which can be the arrangement order of the image to which the target pixel belongs in at least two images). Meanwhile, the adjustment weight of each target pixel can be determined according to the polynomial interpolation principle and the relative position of each pixel position relative to the predetermined image region. Finally, the fusion weight of each target pixel can be determined according to the basic weight of each target pixel and the adjustment weight of each target pixel. For example, if the basic weight is the reciprocal of the denominator (a+b) raised to a certain power, the fusion weight can be obtained by multiplying the basic weight and the adjustment weight. If the basic weight is the denominator (a+b) raised to a certain power, the ratio of the adjustment weight to the basic weight can be taken as the fusion weight.

[0068] For example, the arrangement order of the at least two images is determined according to the shooting direction of the images. For example, the at least two images are four, and the arrangement order of the four images is the first image shot in the left shooting direction, the second image shot in the front shooting direction, the third image shot in the right shooting direction, and the fourth image shot in the rear shooting direction. If the annotation information of the four pixels at a certain pixel position in the four images is 1101 respectively, the pixels to be fused at the certain pixel position include the pixels belonging to the first image, the second image, and the fourth image. According to the polynomial interpolation principle F = w1 (p-1) f1+w1 (p-2) *w2f2+……+w2f N , it can be known that the value of p is 3, and if the certain pixel position is located in the target region 320 described above, the basic weight of the target pixel belonging to the first image can be the reciprocal of the denominator of w1 2 , that is, 1 / (a+b) 2; the base weight for the target pixel belonging to the second image can be the inverse of the denominator of w1*w2, i.e. 1 / (a+b) 2 ; the base weight for the target pixel belonging to the fourth image can be the inverse of the denominator of w2, i.e. 1 / (a+b). Correspondingly, the adjustment weight for the target pixel belonging to the first image can be the numerator of w1 2 , which can be a 2 ; the adjustment weight for the target pixel belonging to the second image can be the numerator of w1*w2, i.e. a*b; the adjustment weight for the target pixel belonging to the fourth image can be the numerator of w2, i.e. b.

[0069] It can be understood that for the other pixels in the at least two pixels except the target pixel, the fusion weight can be set as a second predetermined value, which can be 0 for example.

[0070] It can be understood that the present disclosure can also adopt other principles other than the polynomial interpolation principle to fuse the images, but no matter which fusion principle is adopted, the principle of determining the fusion weight is similar to the principle described above.

[0071] In an embodiment, when determining the adjustment weight, the adjustment weight of only the pixels in the target pixels except the first pixel belonging to the image ranked in the earlier position (which can be referred to as the second pixel) can be determined for example. For example, if the pixels to be fused include the pixels belonging to the first image, the second image and the fourth image, the adjustment weight of only the target pixel belonging to the second image and the target pixel belonging to the fourth image can be determined. Subsequently, the fusion weight of the target pixel belonging to the second image and the fusion weight of the target pixel belonging to the fourth image are calculated to obtain two fusion weights. Finally, the embodiment can determine the fusion weight for the first pixel according to the predetermined sum and the two fusion weights. For example, the difference between the predetermined sum and the sum of the two fusion weights can be taken as the fusion weight for the first pixel. For example, if the fusion weight of the target pixel belonging to the second image is w1*w2 and the fusion weight of the target pixel belonging to the fourth image is w2, the fusion weight of the target pixel belonging to the first image can be represented as (1-w1*w2-w2). By the method of this embodiment, the calculation of the highest power of the value can be avoided, thereby facilitating the reduction of the calculation overhead and further ensuring the implementation of the real-time image fusion technology.

[0072] Figure 4 is a principle diagram for implementing the image fusion method according to an embodiment of the present disclosure.

[0073] As Figure 4 shown, in an embodiment, the image fusion method provided by the present disclosure can be implemented by using an intelligent chip 400. The intelligent chip 400 can be a chip with processing function such as an FPGA chip for example.

[0074] In an embodiment, the intelligent chip 400 can include, for example, a row & column & count module 411, a weight table storage module 412, a zero detection module 413, a weight determination module 414, and a fusion & addition module 415.

[0075] The row & column & count module 411 is configured to count the number of rows, columns, and images of the image. After obtaining the counting result, the row & column & count module 411 generates a predetermined weight table according to the principle shown in FIG. 4B, and stores the predetermined weight table in the weight table storage module 412. Figure 3

[0076] In this embodiment, the number of images that need to be fused can be four. The zero detection module 413 can determine the pixels labeled as 0 according to the labeling information of the first image 401, the second image 402, the third image 403, and the fourth image 404 in response to the input of the four images. The pixels labeled as 0 are the pixels that do not need to be fused. Subsequently, the weight determination module 414 can obtain the base weight from the predetermined weight table stored in the weight table storage module 412 according to the number of pixels that are not labeled as 0 at each pixel position and each pixel position, and calculate the adjustment weight based on the polynomial interpolation principle described above. The fusion weight of each pixel that needs to be fused is calculated according to the base weight and the adjustment weight of each pixel that needs to be fused. Subsequently, the weight determination module 414 can send the fusion weight to the fusion & addition module 415.

[0077] The fusion & addition module 415 is configured to fuse and add the pixels at each pixel position in the first image 401 to the fourth image 404 according to the obtained fusion weight, so as to obtain a fused image and output the fused image.

[0078] In an embodiment, the intelligent chip 400 can further integrate a timing module 416. The timing module 416 is configured to control the timing of outputting the pixel values of each pixel of the fused image by the fusion & addition module 415. For example, the fusion & addition module 415 can be controlled to output the rows and columns of the fused image according to a predetermined timing.

[0079] The embodiments of the present disclosure can realize image fusion based on the intelligent chip 400, so that the image fusion does not need to rely on the host processor, thereby further reducing the occupation of the processing performance of the host processor, and facilitating to improve the operation stability and overall performance of the remote monitoring system of the autonomous vehicle.

[0080] Based on the image fusion method provided by the present disclosure, the present disclosure further provides an image fusion device. The device will be described in detail below. Figure 5

[0081] Figure 5 ​​is a structural block diagram of an image fusion device according to an embodiment of the present disclosure.

[0082] As shown in Figure 5 The image fusion device 500 of this embodiment can include a pixel determining module 510, a first weight determining module 520, and a pixel fusion module 530.

[0083] The pixel determining module 510 is configured to determine pixels of at least two images to be fused at each pixel position, to obtain at least two pixels. In an embodiment, the pixel determining module 510 can be configured to perform the operation S210 described above, and details are not repeated here.

[0084] The first weight determining module 520 is configured to determine fusion weights for the at least two pixels according to a predetermined weight table, each pixel position, and a number of target pixels participating in fusion in the at least two pixels. The predetermined weight table is generated in advance according to the pixel position. In an embodiment, the first weight determining module 520 can be configured to perform the operation S220 described above, and details are not repeated here.

[0085] The pixel fusion module 530 is configured to fuse the at least two pixels according to the fusion weights, to obtain a pixel at each pixel position in a fused image. In an embodiment, the pixel fusion module 530 can be configured to perform the operation S230 described above, and details are not repeated here.

[0086] According to an embodiment of the present disclosure, the first weight determining module 520 described above can include a base weight obtaining sub-module and a fusion weight determining sub-module. The base weight obtaining sub-module is configured to obtain a base weight for each pixel position in the predetermined weight table. The fusion weight determining sub-module is configured to determine the fusion weights for the at least two pixels according to the base weight and the number of target pixels.

[0087] According to an embodiment of the present disclosure, the base weight obtaining sub-module described above can be configured to obtain the base weight for each pixel position in the predetermined weight table in response to each pixel position being located in any of a plurality of predetermined orientations of a predetermined image region. The image fusion device 500 described above can further include a second weight determining module configured to determine the fusion weights for the target pixels as a first predetermined value associated with the number of target pixels in response to each pixel position being located at other positions outside the plurality of predetermined orientations of the predetermined image region.

[0088] According to an embodiment of the present disclosure, the predetermined weight table includes a plurality of sub-tables for the plurality of predetermined orientations. The base weight obtaining sub-module can include a sub-table determining unit and a weight obtaining unit. The sub-table determining unit is configured to determine a target sub-table corresponding to any orientation in the plurality of sub-tables. The weight obtaining unit is configured to obtain the base weight for each pixel position in the target sub-table.

[0089] According to an embodiment of the present disclosure, the fusion weight determination sub-module can include a target weight determination unit, a base weight determination unit, an adjustment weight determination unit, and a fusion weight determination unit. The target weight determination unit is configured to determine target base weights of the target pixels in the base weights according to the number of the target pixels. The base weight determination unit is configured to determine the base weights of each target pixel in the target base weights according to the polynomial interpolation principle and the arrangement order of the target pixels in the at least two pixels. The adjustment weight determination unit is configured to determine the adjustment weights of each target pixel according to the polynomial interpolation principle and the relative positions of each pixel position relative to the predetermined image region. The fusion weight determination unit is configured to determine the fusion weights of each target pixel according to the base weights of each target pixel and the adjustment weights of each target pixel. Wherein, the fusion weights of the pixels other than the target pixels in the at least two pixels are the second predetermined value.

[0090] According to an embodiment of the present disclosure, the target pixels include a plurality of pixels. The adjustment weight determination unit can include a pixel determination sub-unit and an adjustment weight determination sub-unit. The pixel determination sub-unit is configured to determine a first pixel arranged at a front position in the plurality of pixels and a second pixel other than the first pixel according to the arrangement order of the plurality of pixels in the at least two pixels. The adjustment weight determination sub-unit is configured to determine the adjustment weights of each second pixel according to the polynomial interpolation principle and the relative positions. The fusion weight determination unit can include a first determination sub-unit and a second determination sub-unit. The first determination sub-unit is configured to determine the fusion weights of each second pixel according to the adjustment weights of each second pixel and the base weights of each second pixel. The second determination sub-unit is configured to determine the fusion weights of the first pixel according to the predetermined and the fusion weights of all second pixels.

[0091] According to an embodiment of the present disclosure, the image fusion device 500 can further include a position determination module configured to determine that each pixel position is located in any of a plurality of predetermined positions of the predetermined image region, in response to the row where each pixel position is located being outside the row where the predetermined image region is located, and the column where each pixel position is located being outside the column where the predetermined image region is located.

[0092] According to an embodiment of the present disclosure, the predetermined image region includes an area surrounded by two boundary rows and two boundary columns. The plurality of predetermined positions include positions of four target areas surrounded by the two boundary rows, the two boundary columns, and the image boundary. The rows where the four target areas are located are outside the rows where the predetermined image region is located, and the columns where the four target areas are located are outside the columns where the predetermined image region is located.

[0093] According to an embodiment of the present disclosure, for each of the plurality of predetermined orientations, the base weight in the sub-table corresponding to each orientation is determined according to the inverse of the sum of two target distances. The two target distances include: a vertical distance of the pixel position relative to the two boundaries of the predetermined image region that are closest.

[0094] It should be noted that in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information meet the relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs. In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is acquired or collected.

[0095] According to an embodiment of the present disclosure, the present disclosure further provides an intelligent chip, a readable storage medium and a computer program product.

[0096] Figure 6 A schematic block diagram of an example intelligent chip 600 that can be used to implement the image fusion method of the embodiments of the present disclosure is shown. The intelligent chip is integrated in an electronic device, which is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0097] As shown in Figure 6 The intelligent chip 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0098] A plurality of components in the smart chip 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0099] The computing unit 601 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the image fusion method. For example, in some embodiments, the image fusion method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the smart chip 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the image fusion method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the image fusion method by any other appropriate means, such as by means of firmware.

[0100] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0101] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0102] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0103] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0104] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between clients and servers arises by interplay between programs running on the respective computers and having a client-server relationship. Among other things, the server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short) services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0106] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0107] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. An image fusion method, comprising: Determine the pixels at each pixel location of at least two images to be fused, thus obtaining at least two pixels; Based on a predetermined weight table, the position of each pixel, and the number P of target pixels participating in the fusion among the at least two pixels, a fusion weight is determined for the at least two pixels. Specifically, for each target pixel, according to its arrangement order among the at least two pixels, a weight function with exponents (P-1) to 0 in a polynomial is assigned sequentially to the first target pixel to the Pth target pixel in the arrangement order, thereby determining the fusion weight of the target pixel. The fusion weights of the other pixels among the at least two pixels, excluding the target pixel, are set to a second predetermined value. The at least two pixels are fused according to the fusion weights to obtain the pixel at each pixel position in the fused image. The predetermined weight table is generated in advance based on the pixel position.

2. The method according to claim 1, wherein, The step of determining the fusion weights for the at least two pixels based on a predetermined weight table, the position of each pixel, and the number P of target pixels participating in the fusion among the at least two pixels includes: Obtain the base weights for each pixel position from the predetermined weight table; and Based on the base weights and the number of target pixels, a fusion weight is determined for the at least two pixels.

3. The method according to claim 2, wherein: The step of obtaining the base weights for each pixel position in the predetermined weight table includes: in response to each pixel position being located in any of a plurality of predetermined orientations within a predetermined image region, obtaining the base weights for each pixel position in the predetermined weight table; and The method further includes: in response to each pixel location being located at a location other than the plurality of predetermined orientations in the predetermined image region, determining a fusion weight for the target pixel as a first predetermined value associated with the number of target pixels.

4. The method according to claim 3, wherein, The predetermined weight table includes multiple sub-tables for the multiple predetermined directions; Obtaining the base weights for each pixel position from the predetermined weight table includes: Determine the target sub-table corresponding to any of the multiple sub-tables; as well as Obtain the basic weights for each pixel position in the target sub-table.

5. The method according to claim 2, wherein, Determining the fusion weights for the at least two pixels based on the base weights and the number of target pixels includes: Based on the number of target pixels, determine the target base weights for the target pixels in the base weights; Based on the polynomial interpolation principle and the arrangement order of the target pixel among the at least two pixels, the basic weight for each target pixel in the target basic weight is determined; Based on the polynomial interpolation principle and the relative position of each pixel with respect to a predetermined image region, an adjustment weight is determined for each target pixel; and The fusion weight for each target pixel is determined based on the base weight for each target pixel and the adjustment weight for each target pixel.

6. The method according to claim 5, wherein: The target pixel includes multiple pixels; based on the polynomial interpolation principle and the relative position of each pixel with respect to the predetermined image region, the adjustment weight for each target pixel is determined as follows: Based on the arrangement order of the plurality of pixels among the at least two pixels, determine the first pixel that is in the first position among the plurality of pixels and the second pixel other than the first pixel; Based on the polynomial interpolation principle and the relative position, the adjustment weight for each second pixel is determined; The step of determining the fusion weight for each target pixel based on the base weight for each target pixel and the adjustment weight for each target pixel includes: Based on the adjustment weights for each second pixel and the base weights for each second pixel, a fusion weight is determined for each second pixel; and The fusion weight for the first pixel is determined based on the predetermined fusion weight and the fusion weight with all the second pixels.

7. The method according to claim 3, further comprising: In response to the fact that the row containing each pixel position is outside the row containing the predetermined image region, and the column containing each pixel position is outside the column containing the predetermined image region, it is determined that each pixel position is located in any one of a plurality of predetermined orientations within the predetermined image region.

8. The method according to claim 7, wherein: The predetermined image region includes a region enclosed by two boundary rows and two boundary columns; The multiple predetermined orientations include the orientations of the four target regions enclosed by the two boundary rows, the two boundary columns, and the image boundary. Wherein, the row containing the four target regions is outside the row containing the predetermined image region, and the column containing the four target regions is outside the column containing the predetermined image region.

9. The method according to claim 8, wherein, For each of the multiple predetermined directions, the base weight in the sub-table corresponding to each direction is determined based on the reciprocal of the sum of the distances between the two targets. The two target distances include the vertical distance between the pixel positions relative to the two nearest boundaries of the predetermined image region.

10. An image fusion apparatus, comprising: The pixel determination module is used to determine the pixels at each pixel location of at least two images to be fused, thereby obtaining at least two pixels; A first weight determination module is configured to determine a fusion weight for the at least two pixels based on a predetermined weight table, the position of each pixel, and the number P of target pixels participating in the fusion among the at least two pixels. Specifically, for each target pixel, according to its arrangement order among the at least two pixels, a weight function with exponents (P-1) to 0 is sequentially assigned to the first target pixel to the Pth target pixel in the arrangement order, thereby determining the fusion weight of the target pixel. The fusion weights of the other pixels among the at least two pixels, excluding the target pixel, are set to a second predetermined value. A pixel fusion module is used to fuse the at least two pixels according to the fusion weights to obtain the pixel at each pixel position in the fused image. The predetermined weight table is generated in advance based on the pixel position.

11. The apparatus according to claim 10, wherein, The first weight determination module includes: The basic weight acquisition submodule is used to acquire the basic weights for each pixel position in the predetermined weight table; and The fusion weight determination submodule is used to determine the fusion weight for the at least two pixels based on the base weight and the number of target pixels.

12. The apparatus according to claim 11, wherein: The basic weight acquisition submodule is used to: in response to each pixel position being located in any of a plurality of predetermined orientations in a predetermined image region, acquire the basic weight for each pixel position in the predetermined weight table; as well as The apparatus further includes a second weight determination module, configured to determine a fusion weight for the target pixel as a first predetermined value associated with the number of target pixels in response to the fact that each pixel location is located at a location other than the plurality of predetermined orientations in the predetermined image region.

13. The apparatus according to claim 12, wherein, The predetermined weight table includes multiple sub-tables for the multiple predetermined directions; The basic weight acquisition submodule includes: A sub-table determination unit is used to determine the target sub-table corresponding to any of the multiple sub-tables; as well as The weight acquisition unit is used to acquire the basic weights for each pixel position in the target sub-table.

14. The apparatus according to claim 11, wherein, The fusion weight determination submodule includes: The target weight determination unit is used to determine the target base weight for the target pixel in the base weight based on the number of target pixels; The basic weight determination unit is used to determine the basic weight for each target pixel in the target basic weight according to the polynomial interpolation principle and the arrangement order of the target pixel in the at least two pixels; The adjustment weight determination unit is used to determine the adjustment weight for each target pixel based on the polynomial interpolation principle and the relative position of each pixel position with respect to a predetermined image region; and The fusion weight determination unit is used to determine the fusion weight for each target pixel based on the base weight for each target pixel and the adjustment weight for each target pixel.

15. The apparatus according to claim 14, wherein: The target pixel includes multiple pixels; The adjustment weight determination unit includes: A pixel determination subunit is used to determine, based on the arrangement order of the plurality of pixels among the at least two pixels, the first pixel in the first position and the second pixel other than the first pixel; The adjustment weight determination subunit is used to determine the adjustment weight for each second pixel based on the polynomial interpolation principle and the relative position. The fusion weight determination unit includes: A first determining subunit is configured to determine a fusion weight for each second pixel based on an adjustment weight for each second pixel and a base weight for each second pixel; and The second determining subunit is used to determine the fusion weight for the first pixel based on the predetermined fusion weight and the fusion weight with all the second pixels.

16. The apparatus of claim 12, further comprising: The orientation determination module is used to determine, in response to the fact that the row where each pixel position is located is outside the row where the predetermined image region is located, and the column where each pixel position is located is outside the column where the predetermined image region is located, any one of a plurality of predetermined orientations of each pixel position in the predetermined image region.

17. The apparatus according to claim 16, wherein: The predetermined image region includes a region enclosed by two boundary rows and two boundary columns; The multiple predetermined orientations include the orientations of the four target regions enclosed by the two boundary rows, the two boundary columns, and the image boundary. Wherein, the row containing the four target regions is outside the row containing the predetermined image region, and the column containing the four target regions is outside the column containing the predetermined image region.

18. The apparatus according to claim 17, wherein, For each of the multiple predetermined directions, the base weight in the sub-table corresponding to each direction is determined based on the reciprocal of the sum of the distances between the two targets. The two target distances include the vertical distance between the pixel positions relative to the two nearest boundaries of the predetermined image region.

19. A smart chip, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images