Video fusion circuit, method and apparatus, electronic device, computer readable medium

Abstract

This disclosure provides a video fusion method, specifically relating to the technical fields of autonomous driving and image processing. The specific implementation scheme is as follows: Image calibration coordinates and fused image coordinates from multiple fisheye cameras are acquired, along with multiple video streams captured by the fisheye cameras. The fused image coordinates are obtained by fusing all image calibration coordinates. The image calibration coordinates are encoded to obtain coordinate information encoding. Based on the fused image coordinates, weight parameters for the image pixels of each video stream are obtained. Based on the coordinate information encoding, image pixels from each video stream are selected. Based on the weight parameters and image pixels of each video stream, the fused image of all images from the corresponding video streams is calculated. This implementation improves the real-time performance of video fusion.

Description

[0001] This application is a divisional application of the application for "video fusion circuit, method and apparatus, electronic device and computer-readable medium". The original application was filed on December 22, 2022, with application number CN202211666974.4 and invention title: video fusion circuit, method and apparatus, electronic device and computer-readable medium. Technical Field

[0002] This disclosure relates to the field of computer application technology, specifically to the fields of autonomous driving, image processing, etc., and in particular to a video fusion circuit, method and apparatus, electronic device, computer-readable medium and computer program product. Background Technology

[0004] Remote monitoring of autonomous vehicles can eliminate blind spots by fusing multiple video feeds. However, fusing multiple video feeds is based on a central processing unit (CPU). The CPU has limited resources and cannot achieve real-time performance. Furthermore, the limited communication bandwidth and onboard computing resources will lead to a decrease in system performance. Summary of the Invention

[0006] A video fusion circuit, method and apparatus, electronic device, computer-readable storage medium and computer program product are provided.

[0007] According to the first aspect, a video fusion method is provided, which includes: acquiring image calibration coordinates of multiple fisheye cameras, fused image coordinates, and multiple videos captured by multiple fisheye cameras, wherein the fused image coordinates are obtained by fusing all image calibration coordinates; encoding the image calibration coordinates to obtain coordinate information encoding; obtaining weight parameters of image pixels of each image in the multiple videos based on the fused image coordinates; selecting image pixels of each image in the multiple videos based on the coordinate information encoding; and calculating the fused image of all images in the corresponding multiple videos based on the weight parameters of each image and the image pixels of each image.

[0008] According to a second aspect, a video fusion apparatus is provided, comprising: an acquisition unit configured to acquire image calibration coordinates of multiple fisheye cameras, fused image coordinates, and multiple video streams captured by the multiple fisheye cameras, wherein the fused image coordinates are obtained by fusing all image calibration coordinates; an encoding unit configured to encode the image calibration coordinates to obtain coordinate information encoding; an acquisition unit configured to obtain weight parameters of image pixels of each image stream in the multiple video streams based on the fused image coordinates; a selection unit configured to select image pixels of each image stream in the multiple video streams based on the coordinate information encoding; and a calculation unit configured to calculate a fused image of all images in the corresponding multiple video streams based on the weight parameters of each image stream and the image pixels of each image stream.

[0009] According to the third aspect, a video fusion circuit is also provided, comprising: multiple fisheye cameras, a field-programmable gate array (FPGA) circuit, and a synchronous dynamic random access memory (DRAM). The DRAM is used to store multiple video streams captured by the multiple fisheye cameras, as well as the image calibration coordinates and fused image coordinates of the multiple fisheye cameras. The fused image coordinates are obtained by fusing all the image calibration coordinates. The FPGA circuit is used to encode the image calibration coordinates to obtain coordinate information encoding, and stores the coordinate information encoding in the DRAM. Based on the fused image coordinates, the weight parameters of the image pixels of each image in the multiple video streams are obtained, and the weight parameters are stored in the DRAM. Based on the coordinate information encoding, the image pixels of each image in the multiple video streams are selected from the DRAM. Based on the weight parameters and the image pixels of each image, the fused image of all images in the corresponding multiple video streams is calculated.

[0010] According to a fourth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0011] According to a fifth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first aspect.

[0012] According to a sixth aspect, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0013] The video fusion method provided in this disclosure first acquires the image calibration coordinates, fused image coordinates, and multiple video streams captured by multiple fisheye cameras. The fused image coordinates are obtained by fusing all image calibration coordinates. Second, the image calibration coordinates are encoded to obtain coordinate information encoding. Third, based on the fused image coordinates, weight parameters of the image pixels in each video stream are obtained. Fourth, based on the coordinate information encoding, image pixels in each video stream are selected. Finally, based on the weight parameters and image pixels of each video stream, the fused image of all images in the corresponding video stream is calculated. Therefore, by encoding the image calibration coordinates to obtain coordinate information encoding and selecting image pixels based on the coordinate information encoding, the encoding of fusion parameters is optimized, reducing the resource consumption during fusion algorithm processing and improving the real-time effect of video synthesis.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart of an embodiment of the video fusion method according to the present disclosure; Figure 2 This is a schematic diagram of a structure for state transitions in this public code; Figure 3 This is a schematic diagram of a structure according to an embodiment of the video fusion apparatus of this disclosure; Figure 4 This is a schematic diagram of a structure according to an embodiment of the video fusion circuit of this disclosure; Figure 5 This is a schematic diagram of one embodiment of the field-programmable gate array circuit in this disclosure; Figure 6 This is a block diagram of an electronic device used to implement the video fusion method of the embodiments of this disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] In this embodiment, "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0021] This disclosure provides a video fusion method. Figure 1 A flow 100 of an embodiment of the video fusion method according to the present disclosure is shown, the video fusion method comprising the following steps: Step 101: Obtain the image calibration coordinates, fused image coordinates, and multiple video streams captured by the multi-channel fisheye cameras.

[0022] The fused image coordinates are obtained by fusing the calibration coordinates of all images.

[0023] In this embodiment, the image calibration coordinates are obtained by correcting the fisheye image coordinates of the fisheye image captured by the fisheye camera; the fused image is the image obtained by stitching together all the images from multiple video streams, and the fused image coordinates are the coordinates obtained by fusing all the image calibration coordinates. Specifically, all the image calibration coordinates can be transformed into a unified coordinate system. In this coordinate system, the coordinate values ​​of the non-overlapping areas (areas where the calibration coordinates of multiple images do not coincide) are retained. For the coordinate values ​​of the overlapping areas generated by all the image calibration coordinates, the coordinates of the overlapping areas are fused to obtain the fused coordinates; the coordinates of the non-overlapping areas and the fused coordinates of the overlapping areas are combined to obtain the fused image coordinates.

[0024] For example, when fusing four fisheye images, the resulting fused image has a coordinate range of (100×100) and an image calibration coordinate range of (200×200) for each of the four input images. If one fused image coordinate point is (0, 0), it is obtained by fusing the coordinates of all four images together. The image calibration coordinates for the first image are (10, 20), the second image are (30, 100), the third image are (2, 10), and the fourth image are (3, 1). Therefore, the fused image uses the image calibration coordinates of all four images.

[0025] Step 102: Encode the image calibration coordinates to obtain coordinate information encoding.

[0026] In this embodiment, the values ​​of the image calibration coordinates are analyzed to determine the initial coordinate values. The initial coordinate values ​​are then encoded according to a first fixed encoding format to obtain the initial coordinate codes. Based on the difference between the set position image calibration coordinates after the initial coordinate values ​​and the initial coordinate values, the offset is determined. The set position and the offset are then encoded according to a second fixed encoding format to obtain the continuous coordinate codes. The initial coordinate codes and multiple continuous coordinate codes are used as coordinate information encoding.

[0027] Step 103: Based on the fused image coordinates, obtain the weight parameters of the image pixels of each video stream.

[0028] In this embodiment, the fused image is obtained by stitching together the images from multiple video streams. Since the coordinates of the fused image are different from their positions in the overall fused image, the pixel requirements for the fused image obtained by fusing the images from each stream are different. By assigning weight parameters to the image pixels of each stream, the proportion of the image pixels of each stream in the fused image can be determined.

[0029] In this embodiment, the weight parameter is also the proportion of the image pixel values ​​of each image in the fused image. The weight parameter can be a fixed parameter pre-configured based on the coordinates of the fused image. The specific configuration process is as follows: determine the independent region in the coordinates of the fused image (only the image calibration coordinates of one fisheye image are involved when generating the coordinates of the fused image), set the weight parameter of the image pixel of the image corresponding to the independent region to 1, and the image calibration coordinates of the corresponding fisheye image of the image; determine the overlapping region in the coordinates of the fused image (the image calibration coordinates of multiple fisheye images are involved when generating the coordinates of the fused image), set different weight parameters for each image corresponding to the overlapping region, and the sum of the weight parameters of all images corresponding to the overlapping region is 1. Store the correspondence between all weight parameters and the coordinates of the fused image in the weight coordinate correspondence table.

[0030] In this embodiment, the weight parameters of image pixels of each image in multiple videos obtained based on the fused image coordinates include: loading a weight coordinate correspondence table, which is used to characterize the correspondence between the weight parameters and the fused image coordinates.

[0031] In this embodiment, the image calibration coordinates can be recovered by encoding the coordinate information, and the corresponding fused image coordinates can be obtained from the image calibration coordinates. The image calibration coordinates correspond to the pixel values ​​of the image pixels of each video stream. The fused image coordinates correspond to weight parameters. The pixel values ​​of the fused image are obtained by multiplying the weight parameters of each video stream by the pixel values ​​of each video stream.

[0032] Step 104: Based on coordinate information encoding, select image pixels from each video stream.

[0033] In this embodiment, since the coordinate information encoding is information that encodes the image calibration coordinates, the image calibration coordinates can be obtained based on the coordinate information encoding; since the image calibration coordinates are obtained after correcting the pixel coordinates of the fisheye image, the pixel coordinates of the fisheye image can be obtained based on the image calibration coordinates, and the image pixels corresponding to the fisheye image can be queried from the pixel coordinates.

[0034] In this embodiment, image pixels are specific values ​​of pixels. The specific representation of a pixel can be obtained by fusing image coordinates and image pixels.

[0035] Optionally, step 104 above includes: determining image calibration coordinates based on coordinate information encoding; determining image pixels corresponding to the image calibration coordinates based on the image calibration coordinates; and selecting image pixels corresponding to the fused image coordinates.

[0036] Step 105: Based on the weight parameters of each image and the image pixels of each image, calculate the fused image of all images in the corresponding multi-channel video.

[0037] In this embodiment, the weight parameters of each image are multiplied by the image pixels and then added together to obtain the image pixels corresponding to the coordinates of the fused image. The image pixels under all the coordinates of the fused image are combined to obtain the fused image.

[0038] Optionally, after obtaining the fused image, the image can be compressed and uploaded to the cloud for real-time monitoring.

[0039] Compared to traditional methods that consume significant resources for central processing units (CPUs) in fusion algorithm processing, making real-time processing difficult and reducing overall system performance and stability, the video fusion method disclosed in this paper optimizes the coordinate encoding of the fused image. Furthermore, editing the input video based on the coordinates of the fused image reduces system bandwidth. Storing pre-calculated weight parameters accelerates the multi-channel video fusion calculation process. Finally, compressing the fused video and uploading it to the cloud for real-time monitoring improves the real-time effectiveness of fused video surveillance.

[0040] The video fusion method provided in this embodiment first acquires the image calibration coordinates, fused image coordinates, and multiple video streams captured by multiple fisheye cameras. The fused image coordinates are obtained by fusing all image calibration coordinates. Second, the image calibration coordinates are encoded to obtain coordinate information encoding. Third, based on the fused image coordinates, the weight parameters of the image pixels in each video stream are obtained. Fourth, based on the coordinate information encoding, the image pixels of each video stream are selected. Finally, based on the weight parameters and image pixels of each video stream, the fused image of all images in the corresponding video stream is calculated. Therefore, by encoding the image calibration coordinates to obtain coordinate information encoding, and selecting image pixels based on the coordinate information encoding, the encoding of fusion parameters is optimized, reducing the resource consumption of the fusion algorithm and improving the real-time effect of video synthesis.

[0041] In some optional implementations of this embodiment, encoding the image calibration coordinates to obtain coordinate information encoding includes: For each image in a multi-channel video stream, the image calibration coordinates are sequentially input into a state machine to obtain the encoded state output by the state machine. The state machine performs encoding state transitions based on the input image calibration coordinate values ​​and the number of image calibration coordinates. While sequentially inputting the image calibration coordinates into the state machine, any two adjacent image calibration coordinates in the corresponding image stream are compared to obtain the difference between the two image calibration coordinates and their growth direction. Under different encoding states, the image index, the value of the image calibration coordinates, the difference, and the growth direction are encoded according to the corresponding encoding format to obtain the coordinate information encoding.

[0042] In this embodiment, the image index is the identifier of the image in each video stream. The image index can be used to determine which video stream the current image belongs to. By encoding the image index into the coordinate information, the image calibration coordinates corresponding to each image can be quickly found through the coordinate information encoding. For example, the multi-video stream consists of four video streams, and each video stream has the same image index. The image indices of the four video streams are 1 to 4.

[0043] In this embodiment, a state machine generates encoded states, and coordinate information encoding corresponding to each encoded state is obtained. The coordinate information encoding is different for each encoded state, and each encoded state can have one or multiple encoding formats. For example... Figure 2 As shown, the state machine has four encoding states: idle coordinate state Si, initial coordinate state S0, first coordinate state S1, and continuous coordinate encoding state S2. For different encoding states, the corresponding coordinate information encoding can be obtained.

[0044] image_cnt is used to count the number of input image calibration coordinates I, image_cnt <= NUM (NUM is the maximum number of coordinates per image), and image_index is used to indicate the index of the current image being processed, image_index <= n (n is the number of fisheye cameras).

[0045] Idle coordinate state Si: Process according to image_index. If image_index == n, it indicates that the encoding corresponding to n images has been processed. Otherwise, issue the specific image encoding according to image_index. Make a state transition judgment when the input image calibration coordinate I of each image is valid. When the value of the image calibration coordinate I is 0, that is, when the Ci0 condition is satisfied, jump to the initial coordinate state S0; otherwise, when the Ci0 condition is not satisfied, jump to the first coordinate state S1.

[0046] Initial coordinate state S0: Use zero_num to count the number of 0 coordinates in the initial coordinate state S0.

[0047] If the value of the input image calibration coordinate I is 0 and image_cnt == NUM, when the C0i condition is satisfied, jump back to the idle coordinate state Si. If the value of the input image calibration coordinate I is not 0 and image_cnt == NUM, when the C0i condition is satisfied, jump back to the idle coordinate state Si and set a non-zero last flag bit at the same time.

[0048] If the value of the input image calibration coordinate I is 0 and image_cn < NUM, zero_num is incremented by 1 and remains in the initial coordinate state S0. If the value of the input image calibration coordinate I is not 0 and image_cn < NUM, when the C01 condition is satisfied, jump to the first coordinate state S1.

[0049] First coordinate state S1: In the first coordinate state S1, it indicates that the current is the first valid pixel coordinate point.

[0050] If the value of the input image calibration coordinate I is 0 and image_cnt == NUM, when the C1i condition is satisfied, jump back to the idle coordinate state Si and set a zero last flag bit at the same time. ]

[0051] If the value of the input image calibration coordinate I is not 0 and image_cnt == NUM, when the C1i condition is satisfied, jump back to the idle coordinate state Si and set a non-zero last flag bit at the same time.

[0052] If the value of the input image calibration coordinate I is 0 and image_cnt < NUM, and C10 condition is satisfied, then jump back to the initial coordinate state S0.

[0053] If the value of the input image calibration coordinate I is not 0 and image_cnt < NUM, and C12 condition is satisfied, then jump back to the continuous coordinate encoding state S2 to prepare for continuous encoding.

[0054] Continuous coordinate encoding state S2: In the continuous coordinate encoding state S2, it indicates that the current enters the continuous pixel point encoding state.

[0055] If the value of the input image calibration coordinate I is 0 and image_cnt == NUM, and C2i condition is satisfied, then jump back to the idle coordinate state Si, and at the same time set a last flag bit zero state.

[0056] If the value of the input image calibration coordinate I is not 0 and image_cnt == NUM, and C2i condition is satisfied, then jump back to the idle coordinate state Si, and at the same time set a last flag bit non - zero state.

[0057] If the value of the input image calibration coordinate I is 0 and image_cnt < NUM, and C20 condition is satisfied, then jump back to the initial coordinate state S0.

[0058] If the value of the input image calibration coordinate I is not 0 and image_cnt < NUM, it is necessary to judge the difference diff between the previous and the next 2 pixels. If the difference diff < 15, continue to stay in the continuous coordinate encoding state S2. Otherwise, when diff >= 15, then C21 condition is satisfied, and it is necessary to jump to the first coordinate state S1 to start new encoding.

[0059] In the idle coordinate state Si, if it is found that the last flag bit non - zero state is true, then take the input image calibration coordinate I at this moment as the starting address to obtain pixel points. If the last flag bit is zero state, then consider the last pixel point as 0.

[0060] In this embodiment, in the continuous coordinate encoding state S2, the difference diff is the difference between two adjacent input coordinates. The growth direction is used to indicate whether it is positive growth or reverse growth between two adjacent input coordinates. In this embodiment, the input coordinate is the input image calibration coordinate.

[0061] The optional implementation provides a method for encoding image calibration coordinates. The image calibration coordinates are sequentially input into a state machine, causing the state machine to transition between encoding states based on the values ​​and quantity of the image calibration coordinates, resulting in at least one encoding state. Simultaneously with the sequential input of the image calibration coordinates into the state machine, any two adjacent image calibration coordinates in the image path are compared to obtain the difference and growth direction of the two coordinates. Under different encoding states, the image index, the values ​​of the image calibration coordinates, and the growth direction are encoded according to the corresponding encoding format to obtain the coordinate information encoding. This provides a reliable implementation method for image calibration coordinate encoding.

[0063] In some optional implementations of this embodiment, the encoding state includes at least one of the following: an initial coordinate state, a first coordinate state, and a continuous coordinate encoding state, arranged sequentially. Under different encoding states, the image index, the value of the image calibration coordinates, the difference, and the growth direction are encoded according to the corresponding encoding format to obtain coordinate information encoding. This includes: in the initial coordinate state, in response to the input coordinate value being zero, obtaining the number of zero coordinates output by the state machine, and encoding the number and the image index according to the first encoding format to obtain zero-point pixel coordinate encoding; in the first coordinate state, in response to the input coordinate value not being zero, encoding the input coordinate value and the image index according to the second encoding format to obtain initial pixel coordinate encoding; in the continuous coordinate state, in response to the input coordinate value not being zero, encoding the image index, the difference, and the growth direction according to the third encoding format to obtain continuous pixel coordinate encoding.

[0064] In this embodiment, the initial coordinate state is: If the input image calibration coordinate value is 0 and image_cnt==NUM, the number zero_num and the image index are encoded and combined according to the first encoding format to obtain the zero-point pixel coordinate encoding, as shown in Table 1.

[0065] Table 1

[0066] When Encode = 0: it indicates that the current pixel coordinate is 0. zero_num is a consecutive length of 0.

[0067] Besides the format shown in Table 1, the first encoding format can also be other formats. For example, if the input image calibration coordinate value is not 0 and image_cnt==NUM, two zero-point pixel coordinate codes will be output: 1. Set encode=0, combine zero_num and encode to obtain one zero-point pixel coordinate code; 2. Set encode=F, index=1, combine the input image calibration coordinate value with encode and index to output the other zero-point pixel coordinate code.

[0068] In the first coordinate state: In response to the input coordinates being non-zero, the input coordinates (XAddr, YAddr) and the image index are encoded according to the second encoding format to obtain the initial pixel coordinate encoding, as shown in Table 2.

[0069] Table 2

[0070] When Encode = 1: Only one pixel is valid, and XAddr and YAddr are the initial addresses of the pixel in the horizontal and vertical directions.

[0071] Optionally, if the input coordinate value is 0 and image_cnt==NUM, two initial pixel coordinate codes will be output: 1. Set encode=1, index=1, and combine XADDR, YADDR, encode, and index to output one initial pixel coordinate code. 2. Set encode=F, index=F, zero_num=1, and combine encode, index, and zero_num to output another initial pixel coordinate code.

[0072] In continuous coordinate encoding state: In response to the input coordinates being non-zero, the image index, difference, and growth direction are encoded according to the third encoding format to obtain the continuous pixel coordinate encoding, as shown in Table 3.

[0073] Table 3

[0074] When Encode = 2: There are more than 1 valid pixels, and XAddr and YAddr are the initial addresses of the pixels in the horizontal and vertical directions.

[0075] When Encode = 3: it is a subset of Encode = 2, with offset 1, offset 2... offset6 being the difference between two adjacent pixels. direction indicates whether the increase between two adjacent pixels is forward or backward.

[0076] In this optional implementation, zero-point pixel coordinate encoding, initial pixel coordinate encoding, and continuous pixel coordinate encoding belong to coordinate information encoding. The encoding format corresponding to the coordinate information encoding is different under different encoding states. Therefore, a diversified coordinate information encoding that can express different content is provided.

[0077] The optional implementation provides a method for obtaining coordinate information encoding, offering different encoding formats for coordinate information encoding under different encoding states, thus providing a reliable basis for effectively representing image calibration coordinates.

[0078] Optionally, the above-mentioned encoding state also includes: an idle coordinate state preceding the initial coordinate state, such as... Figure 2 In the above, the idle coordinate state Si is a state that determines the image index. Under different encoding states, the image index, the value of the image calibration coordinate, the difference, and the growth direction are encoded according to the corresponding encoding format to obtain coordinate information encoding. This includes: in the idle coordinate state, determining the image index so that the image index can participate in the coordinate information encoding of other coordinate states.

[0079] In some optional implementations of this embodiment, the state machine is further used to record the initial image calibration coordinates and the final image calibration coordinates in each round of the first coordinate state; the second encoding format includes: a first sub-format and a second sub-format. In the first coordinate state, in response to the input coordinate value not being zero, the input coordinate value and the image index are encoded according to the second encoding format to obtain the initial pixel coordinate encoding, including: in response to the input coordinate value not being zero and the number of image calibration coordinates of the path image reaching a preset number, the input coordinate value and the image index are encoded according to the first sub-format to obtain the first initial coordinate encoding; in response to the input coordinate value not being zero and the number of image calibration coordinates of the path image not reaching the preset number, the difference between the final image calibration coordinate and the initial image calibration coordinate is calculated to obtain the continuous length; the image index, the continuous length, and the image calibration coordinate value are encoded according to the second sub-format to obtain the second initial coordinate encoding.

[0080] In this optional implementation, the first sub-format and the second sub-format are determined based on the specific content of the generated initial pixel coordinate encoding. When this content differs, the first and second sub-formats can be adaptively improved based on requirements. The preset quantity is the maximum number of coordinates for each image channel; for example, the coordinates of one image channel are 100×100.

[0081] In this optional implementation, the initial pixel coordinate encoding includes: a first initial coordinate encoding and a second initial coordinate encoding. The first initial coordinate encoding is the encoding state after the image calibration coordinates of each image channel reach a preset number of image coordinates; the second initial coordinate encoding is the encoding state after the image calibration coordinates of each image channel have not reached the preset number. At this time, recording the continuous length of the overall encoding can facilitate the determination of the position reached by the encoding.

[0082] This optional implementation provides an initial pixel coordinate encoding. In the first coordinate state of each round, the state machine records the initial and final image calibration coordinates, which can provide information support for the encoding of multiple coordinates in subsequent consecutive coordinate states, ensuring the comprehensiveness of the coordinate information encoding.

[0084] In some optional implementations of this disclosure, the above-mentioned method of obtaining the weight parameters of the image pixels of each video in multiple video streams based on the fused image coordinates includes: setting a first balance variable and a second balance variable, wherein the first balance variable and the second balance variable are dynamically adjusted based on the size of the fused image coordinates; and, in response to the field of view of the multiple fisheye cameras being used to cover the entire area of ​​the vehicle, calculating the weight parameters of the image pixels of each video stream based on the first balance variable and the second balance variable respectively.

[0085] In a specific example of this disclosure, the multi-channel fisheye camera is a four-channel fisheye camera. In the direction of looking down at the vehicle, the four fisheye cameras are a left fisheye camera monitoring the left side area of ​​the vehicle, an upper fisheye camera monitoring the upper side area of ​​the vehicle, a lower fisheye camera monitoring the lower side area of ​​the vehicle, and a right fisheye camera monitoring the right side area of ​​the vehicle.

[0086] For the left fused image coordinates (i1, j1) obtained from the image calibration coordinates of the left fisheye camera, the weight parameters corresponding to the left fused image coordinates (i1, j1) are: ,in, It is the first equilibrium variable. It is the second equilibrium variable.

[0087] For the upper fused image coordinates (i2, j2) obtained from the image calibration coordinates of the upper fisheye camera in the fused image coordinates, the weight parameters corresponding to the upper fused image coordinates (i2, j2) are: .

[0088] For the lower fused image coordinates (i3, j3) obtained from the image calibration coordinates of the lower fisheye camera, the weight parameters corresponding to the lower fused image coordinates (i3, j3) are: .

[0089] For the right fused image coordinates (i4, j4) obtained from the image calibration coordinates of the right fisheye camera, the weight parameters corresponding to the right fused image coordinates (i4, j4) are: .

[0090] In this embodiment, both the first balancing variable and the second balancing variable are variables with a certain range. The execution entity running on the video fusion method dynamically adjusts the first balancing variable and the second balancing variable according to the size of the fused image coordinates, thereby maintaining the pixel fusion of each image of each video within a certain range and ensuring the reliability of pixel display.

[0091] In some optional implementations of this disclosure, the acquisition of image calibration coordinates, fused image coordinates, and multiple videos captured by the multiple fisheye cameras includes: acquiring videos from each fisheye camera in the multiple fisheye cameras, performing boundary clipping on the videos, and decomposing the clipped videos to obtain images from each of the multiple videos; acquiring the fisheye image coordinates of the images from the multiple cameras, correcting the fisheye image coordinates to obtain the image calibration coordinates of each image in the multiple fisheye cameras; and determining the fused image coordinates based on all the image calibration coordinates.

[0092] In this optional implementation, the execution entity running on the video fusion method determines the maximum and minimum values ​​of the pixel values ​​in the row and vertical directions of the image in each video. When the pixel value is the minimum, it is determined as the boundary point of each video. The boundary point of each video is found, and the image is cropped according to this boundary point. Furthermore, the images of each video are edited based on the boundary of each video and the cropped images of each video are decomposed using decomposition software, and cached in DDR (Double Data Rate, Double Data Rate Synchronous Dynamic Random Memory).

[0093] In this optional implementation, after obtaining the images corresponding to each video stream, the parameters of the fisheye camera that captured each image can be obtained. Based on the parameters, the camera model of the fisheye camera is determined. Using the camera model of the fisheye camera, the mapping relationship between the pixel coordinates of the normal image and the distorted image is found. The fisheye image coordinates of the fisheye image are corrected using this mapping relationship to obtain the image calibration coordinates of the multiple fisheye cameras.

[0094] By employing a certain matching strategy, the positions of templates or image feature points in all images of the multi-channel video are found in the reference image (any one of the images in the multi-channel video can be used as the reference image), thereby determining the transformation relationship between all images of the multi-channel video.

[0095] Based on the correspondence between templates or image feature points, the parameter values ​​in the mathematical model are calculated, thereby establishing a mathematical transformation model for all images in the multi-channel video. Based on the established mathematical transformation model, the image calibration coordinates of the multi-channel fisheye cameras are transformed into the coordinate system of the reference image, completing the unified coordinate transformation, and obtaining the fused image coordinates of all images in the multi-channel fisheye cameras, as well as the correspondence between the image calibration coordinates and the fused image coordinates.

[0096] In this embodiment, the fused image is the image obtained by stitching together all the images from multiple video streams. Specifically, a stitching algorithm can be used to stitch together all the images. Image stitching algorithms are mature algorithms and will not be described in detail here.

[0097] Optionally, after obtaining the fused image, the pixel coordinates of each pixel in the fused image are collected to obtain the coordinates of the fused image.

[0098] The optional implementation provides a method for obtaining image calibration coordinates, fused image coordinates, and multiple video streams, which can effectively obtain image calibration coordinates, fused image coordinates, and multiple video streams, improving the reliability of information acquisition.

[0100] Further reference Figure 3 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a video fusion apparatus, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0101] like Figure 3 As shown, the video fusion apparatus 300 provided in this embodiment includes: an acquisition unit 301, an encoding unit 302, a obtaining unit 303, a selection unit 304, and a calculation unit 305. The acquisition unit 301 can be configured to acquire image calibration coordinates of multiple fisheye cameras, fused image coordinates, and multiple video streams captured by the multiple fisheye cameras. The fused image coordinates are obtained by fusing all image calibration coordinates. The encoding unit 302 can be configured to encode the image calibration coordinates to obtain coordinate information encoding. The obtaining unit 303 can be configured to obtain the weight parameters of image pixels of each image in the multiple video streams based on the fused image coordinates. The selection unit 304 can be configured to select image pixels of each image in the multiple video streams based on the coordinate information encoding. The calculation unit 305 can be configured to calculate the fused image of all images in the corresponding multiple video streams based on the weight parameters and image pixels of each image.

[0102] In this embodiment, the specific processing of the video fusion device 300, including the acquisition unit 301, encoding unit 302, obtaining unit 303, selection unit 304, and calculation unit 305, and the resulting technical effects, can be found in reference [reference needed]. Figure 1 The relevant descriptions of steps 101, 102, 103, 104, and 105 in the corresponding embodiments will not be repeated here.

[0103] In some optional implementations of this disclosure, the encoding unit 302 is further configured to: sequentially input image calibration coordinates into a state machine for each image in the multi-channel video, obtain the encoding state output by the state machine, and perform encoding state transitions based on the input image calibration coordinate values ​​and the number of image calibration coordinates; while sequentially inputting the image calibration coordinates into the state machine, compare any two adjacent image calibration coordinates in the image to obtain the difference and growth direction of the two image calibration coordinates; and in different encoding states, encode the image index, the value of the image calibration coordinates, the difference, and the growth direction according to the corresponding encoding format to obtain coordinate information encoding.

[0104] In some optional implementations of this disclosure, the encoding state includes at least one of the following: an initial coordinate state, a first coordinate state, and a continuous coordinate encoding state arranged in sequence; the encoding unit 302 is further configured to: in the initial coordinate state, in response to the value of the input coordinate being zero, obtain the number of zero coordinates output by the state machine, and encode the number and image index according to a first encoding format to obtain the zero-point pixel coordinate encoding; in the first coordinate state, in response to the value of the input coordinate being non-zero, encode the value of the input coordinate and the image index according to a second encoding format to obtain the initial pixel coordinate encoding; in the continuous coordinate state, in response to the value of the input coordinate being non-zero, encode the image index, the difference, and the growth direction according to a third encoding format to obtain the continuous pixel coordinate encoding.

[0105] In some optional implementations of this disclosure, the state machine is further used to record the initial and final image calibration coordinates in each round of the first coordinate state; the second encoding format includes a first sub-format and a second sub-format, and the encoding unit 302 is further configured to: in response to the input coordinate value being non-zero and the number of image calibration coordinates of the path image reaching a preset number, encode the input coordinate value and image index according to the first sub-format to obtain a first initial coordinate code; in response to the input coordinate value being non-zero and the number of image calibration coordinates of the path image not reaching a preset number, calculate the difference between the final image calibration coordinate and the initial image calibration coordinate to obtain a continuous length; encode the image index, continuous length, and image calibration coordinate value according to the second sub-format to obtain a second initial coordinate code.

[0106] In some optional implementations of this disclosure, the above-mentioned obtaining unit 303 is further configured to: set a first balancing variable and a second balancing variable, the first balancing variable and the second balancing variable being dynamically adjusted based on the size of the fused image coordinates; in response to the field of view of the multi-channel fisheye camera being used to cover the entire area of ​​the vehicle, calculate the weight parameters of the image pixels of each channel image based on the first balancing variable and the second balancing variable respectively.

[0107] In some optional implementations of this disclosure, the acquisition unit 301 is further configured to: acquire videos from each fisheye camera in the multi-channel fisheye camera, perform boundary clipping on the videos, and decompose the clipped videos to obtain images from each channel of the multi-channel video; acquire the fisheye image coordinates of the images from the multi-channel cameras, correct the fisheye image coordinates to obtain the image calibration coordinates of each image in the multi-channel fisheye camera; and determine the fused image coordinates based on all image calibration coordinates.

[0108] The video fusion apparatus provided in the embodiments of this disclosure firstly acquires the image calibration coordinates, fused image coordinates, and multiple video streams captured by multiple fisheye cameras using an acquisition unit 301. The fused image coordinates are obtained by fusing all image calibration coordinates. Secondly, an encoding unit 302 encodes the image calibration coordinates to obtain coordinate information encoding. Thirdly, an acquisition unit 303 obtains the weight parameters of the image pixels of each image in the multiple video streams based on the fused image coordinates. Fourthly, a selection unit 304 selects the image pixels of each image in the multiple video streams based on the coordinate information encoding. Finally, a calculation unit 305 calculates the fused image of all images in the corresponding multiple video streams based on the weight parameters and image pixels of each image. Thus, by encoding the image calibration coordinates to obtain coordinate information encoding, and selecting image pixels based on the coordinate information encoding, the encoding of fusion parameters is optimized, reducing the resource consumption during fusion algorithm processing and improving the real-time effect of video synthesis.

[0109] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure further provides an embodiment of a video fusion circuit, which is similar to... Figure 1 Corresponding to the method embodiment shown, this circuit can be specifically applied to various electronic devices.

[0112] like Figure 4As shown, the video fusion circuit 400 provided in this embodiment includes: a multi-channel fisheye camera 401, a field-programmable gate array (FPGA) circuit 402, and a synchronous dynamic random access memory (DRAM) 403. The DRAM 403 is connected to the FPGA circuit 402, and the multi-channel fisheye camera 401 is electrically connected to the FPGA circuit 402.

[0113] In this embodiment, the field-programmable gate array circuit is implemented using an FPGA (Field-Programmable Gate Array).

[0114] In this embodiment, the synchronous dynamic random access memory 403 is used to store multiple video streams captured by multiple fisheye cameras, as well as the image calibration coordinates and fused image coordinates of the multiple fisheye cameras. The fused image coordinates are obtained by fusing all the image calibration coordinates.

[0115] The field-programmable gate array circuit 402 is used to encode the image calibration coordinates to obtain coordinate information encoding, and store the coordinate information encoding in synchronous dynamic random access memory; based on the fused image coordinates, the weight parameters of the image pixels of each image in the multi-channel video are obtained, and the weight parameters are stored in synchronous dynamic random access memory 403; based on the coordinate information encoding, the image pixels of each image in the multi-channel video are selected from synchronous dynamic random access memory 403; based on the weight parameters of each image and the image pixels of each image, the fused image of all images in the corresponding multi-channel video is calculated.

[0116] In this embodiment, the multi-channel fisheye camera 401 includes at least two fisheye cameras, each fisheye camera captures one video, each video includes multiple frames of images, and each video image has calibrated image calibration coordinates; after fusing the image calibration coordinates of all video images captured by the multi-channel fisheye cameras into a unified coordinate system, fused image coordinates are obtained.

[0117] In this embodiment, the multiple video streams captured by the multi-channel fisheye camera stored in the synchronous dynamic random access memory 403 can be videos processed by the field-programmable gate array circuit 402. Specifically, the field-programmable gate array circuit 402 clips the input video according to the boundary of each input video channel in the multiple video streams and caches it in the synchronous dynamic random access memory 403.

[0118] In this embodiment, the coordinate information encoding corresponds to the image calibration coordinates. Partial or all of the image calibration coordinates can be obtained through coordinate information encoding, and the pixel value corresponding to the image calibration coordinates can be obtained from the image calibration coordinates. The fused image coordinates correspond to the image calibration coordinates. For example, in a multi-channel video stream consisting of four channels, a certain fused image coordinate is obtained by fusing the image calibration coordinates of the four channels, or the fused image coordinates are obtained by fusing the image calibration coordinates of three of the four channels. Multiplying the pixel value by the weight parameter yields the pixel corresponding to the fused image coordinates, thus obtaining the fused image.

[0119] In this embodiment, the field-programmable gate array circuit buffers the video data of multiple video streams into the FIFO (First Input First Output) corresponding to each video stream. Then, according to the instructions of the state machine, it reads the image pixels from the FIFO and retrieves the pre-calculated weight parameters from the LUT (Look-Up Table) to calculate and generate the final fused image.

[0120] In this embodiment, the specific processing of the field-programmable gate array circuit 402 and its resulting technical effects can be found in the following references. Figure 1 The relevant descriptions of steps 101, 102, 103, 104, and 105 in the corresponding embodiments will not be repeated here.

[0121] Among the alternative implementations disclosed herein, such as Figure 5 As shown, the above-mentioned field-programmable gate array circuit includes: a first buffer and a second buffer, a comparator, a state machine, and an encoding circuit; the first buffer and the second buffer respectively store two adjacent image calibration coordinates; the comparator is used to compare any two adjacent image calibration coordinates in the image path to obtain the difference and growth direction of the two image calibration coordinates; the state machine performs encoding state transitions based on the input image calibration coordinate values ​​and the number of image calibration coordinates to obtain different encoding states; the encoding circuit encodes the image index, the image calibration coordinate values, the difference, and the growth direction according to the corresponding encoding format in different encoding states to obtain coordinate information encoding.

[0122] In this embodiment, please refer to the detailed schematic diagram of the state machine's state changes. Figure 2 As shown, under different encoding states provided by the state machine, the encoding circuit encodes the image index, image calibration coordinate values, differences, and growth direction in a fixed encoding format to obtain coordinate information encoding.

[0123] The video fusion circuit provided in the embodiments of this disclosure includes a multi-channel fisheye camera 401 connected to a synchronous dynamic random access memory (DRAM) 403, and a field-programmable gate array (FPGA) 402 connected to the DRAM 403. The FPGA 402 encodes the image calibration coordinates to obtain coordinate information encoding and stores the coordinate information encoding in the DRAM 403. Based on the fused image coordinates, it obtains the weight parameters of the image pixels of each image in the multi-channel video and stores the weight parameters in the DRAM 403. Based on the coordinate information encoding, it selects the image pixels of each image in the multi-channel video from the DRAM 403. Based on the weight parameters and image pixels of each image, it calculates the fused image of all images in the corresponding multi-channel video. Thus, by encoding the image calibration coordinates to obtain coordinate information encoding and selecting image pixels based on the coordinate information encoding, it optimizes the encoding of fusion parameters, reduces the resource consumption during fusion algorithm processing, and improves the real-time effect of video synthesis.

[0124] Optionally, the field-programmable gate array circuit can also determine the coordinate information encoding where the image calibration coordinates are zero, and find the boundary values ​​of each input video based on the coordinate information encoding to buffer the video data of the valid multi-channel video.

[0125] Optionally, the field-programmable gate array (FPGA) can calculate weight parameters based on the fused image resolution parameters in the calibration parameter file and store them in the FPGA's level unit (LUT). The FPGA then reads valid data from the synchronous dynamic random access memory (DRAM) based on the coordinate information encoding and performs calculations using the corresponding parameters in the LUT to generate the fused image.

[0126] Optionally, the fused image can be transmitted to the video compression module in the field-programmable gate array circuit for compression, and then transmitted to the cloud via the network for remote monitoring.

[0127] The video fusion circuit disclosed herein generates image calibration coordinates and fused image coordinates offline and encodes them in a field-programmable gate array (FPGA) circuit to improve the efficiency of the FPGA circuit accessing synchronous dynamic random access memory (DRAM). During the encoding process, boundary values ​​for each input video can be found to buffer valid input video data.

[0128] Compared to traditional methods that consume significant resources for fusion algorithm processing on a central processing unit (CPU), making real-time processing difficult and reducing overall system performance and stability, the video fusion circuit disclosed in this paper optimizes the encoding of fusion parameters on a field-programmable gate array (FPGA). Simultaneously, the input video is edited according to the parameters to reduce the bandwidth of the synchronous dynamic random access memory (DRAM). Furthermore, pre-calculated fusion weight parameters are stored in the FPGA's level unit (LUT) to accelerate the multi-channel input video fusion calculation process. Finally, the fused video is compressed and uploaded to the cloud for real-time monitoring.

[0129] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0131] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0132] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0133] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0134] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as video fusion methods. For example, in some embodiments, the video fusion method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the video fusion method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the video fusion method by any other suitable means (e.g., by means of firmware).

[0136] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable video fusion apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video fusion method, the method comprising: The image calibration coordinates, fused image coordinates, and multiple video streams captured by the multi-channel fisheye cameras are obtained. The fused image coordinates are obtained by fusing all the image calibration coordinates. Based on the fused image coordinates, the weight parameters of the image pixels of each image in the multi-channel video are obtained; Based on the image calibration coordinates, select the image pixels of each image in the multi-channel video; Based on the weight parameters and image pixels of each image, a fused image of all images in the multi-video stream is calculated. The step of obtaining the weight parameters of image pixels in each of the multiple video streams based on the fused image coordinates includes: A first balancing variable and a second balancing variable are set, and the first balancing variable and the second balancing variable are dynamically adjusted based on the magnitude of the coordinates of the fused image; In response to the field of view of the multi-channel fisheye camera being used to cover the entire area of ​​the vehicle, the weight parameters of the image pixels of each channel image are calculated based on the first balance variable and the second balance variable.

2. The method according to claim 1, wherein, The step of selecting image pixels from each of the multiple video streams based on the image calibration coordinates includes: The image calibration coordinates are encoded to obtain coordinate information encoding; Image pixels of each video stream are selected based on the coordinate information encoded.

3. The method according to claim 2, wherein, The process of encoding the image calibration coordinates to obtain coordinate information encoding includes: For each image in the multi-channel video, the image calibration coordinates are sequentially input into the state machine to obtain the encoding state output by the state machine. The state machine performs encoding state transitions based on the input image calibration coordinate values ​​and the number of image calibration coordinates. While the image calibration coordinates are sequentially input into the state machine, any two adjacent image calibration coordinates in the image path are compared to obtain the difference between the two image calibration coordinates and the direction of increase. Under different encoding states, the image index, the image calibration coordinate values, the difference, and the growth direction are encoded according to the corresponding encoding format to obtain coordinate information encoding.

4. The method according to claim 3, wherein, The encoding state includes at least one of the following: an initial coordinate state, a first coordinate state, and a continuous coordinate encoding state ordered sequentially. The process of encoding the image index, image calibration coordinate values, the difference, and the growth direction according to the corresponding encoding format under different encoding states to obtain coordinate information encoding includes: In the initial coordinate state, in response to the value of the input coordinate being zero, the number of zero coordinates output by the state machine is obtained, and the number and image index are encoded according to the first encoding format to obtain the zero-point pixel coordinate encoding; In the first coordinate state, in response to the input coordinate value not being zero, the input coordinate value and image index are encoded according to the second encoding format to obtain the initial pixel coordinate encoding; In the continuous coordinate state, in response to the input coordinate value not being zero, the image index, the difference, and the growth direction are encoded according to the third encoding format to obtain continuous pixel coordinate encoding.

5. The method according to claim 4, wherein, The state machine is also used to record the initial and final image calibration coordinates in each round of the first coordinate state; the second encoding format includes: a first sub-format and a second sub-format, wherein in the first coordinate state, in response to the input coordinate value not being zero, the input coordinate value and image index are encoded according to the second encoding format to obtain the initial pixel coordinate encoding, including: In response to the fact that the value of the input coordinate is not zero and the number of image calibration coordinates of the image path reaches a preset number, the value of the input coordinate and the image index are encoded according to the first sub-format to obtain the first initial coordinate code; In response to the fact that the value of the input coordinate is not zero and the number of image calibration coordinates of the image path has not reached the preset number, the difference between the last image calibration coordinate and the initial image calibration coordinate is calculated to obtain the continuous length. The image index, the continuous length, and the image calibration coordinates are encoded according to the second sub-format to obtain the second initial coordinate code.

6. The method according to any one of claims 1-5, wherein, The acquisition of image calibration coordinates, fused image coordinates, and multiple video streams captured by the multi-channel fisheye cameras includes: Videos from each of the multiple fisheye cameras are acquired separately, the videos are edge-trimmed, and the trimmed videos are decomposed to obtain images from each of the multiple video streams. Obtain the fisheye image coordinates of images from multiple cameras, correct the fisheye image coordinates, and obtain the image calibration coordinates of each image in the multiple fisheye cameras. The coordinates of the fused image are determined based on the calibration coordinates of all images.

7. A video fusion apparatus, the apparatus comprising: The acquisition unit is configured to acquire the image calibration coordinates, fused image coordinates, and multiple video streams captured by the multi-channel fisheye cameras, wherein the fused image coordinates are obtained by fusing all the image calibration coordinates. The unit is configured to obtain the weight parameters of the image pixels of each image in the multi-channel video based on the fused image coordinates; The selection unit is configured to select image pixels of each image in the multi-channel video based on the image calibration coordinates. The calculation unit is configured to calculate a fused image corresponding to all images in the multi-channel video based on the weight parameters of each image and the image pixels of each image. The obtaining unit is further configured to: set a first balance variable and a second balance variable, the first balance variable and the second balance variable being dynamically adjusted based on the size of the fused image coordinates; and, in response to the field of view of the multi-channel fisheye camera being used to cover the entire area of ​​the vehicle, calculate the weight parameters of the image pixels of each channel image based on the first balance variable and the second balance variable.

8. The apparatus according to claim 7, wherein, The selection unit includes: An encoding subunit is configured to encode the image calibration coordinates to obtain coordinate information encoding; The selected sub-unit is configured to select image pixels of each image in the multi-channel video based on the coordinate information encoding.

9. The apparatus according to claim 8, wherein, The encoding subunit is further configured to: for each image in the multi-channel video, sequentially input the image calibration coordinates into the state machine to obtain the encoding state output by the state machine, and the state machine performs encoding state transitions based on the input image calibration coordinate values ​​and the number of image calibration coordinates; While the image calibration coordinates are sequentially input into the state machine, any two adjacent image calibration coordinates in the image path are compared to obtain the difference between the two image calibration coordinates and the direction of increase. Under different encoding states, the image index, the value of the image calibration coordinates, the difference, and the direction of increase are encoded according to the corresponding encoding format to obtain the coordinate information encoding.

10. The apparatus according to claim 9, wherein, The encoding states include at least one of the following: an initial coordinate state, a first coordinate state, and a continuous coordinate encoding state, ordered sequentially; the encoding subunit is further configured to: in the initial coordinate state, in response to the input coordinate value being zero, obtain the number of zero coordinates output by the state machine, and encode the number and image index according to a first encoding format to obtain zero-point pixel coordinate encoding; in the first coordinate state, in response to the input coordinate value not being zero, encode the input coordinate value and image index according to a second encoding format to obtain initial pixel coordinate encoding; in the continuous coordinate state, in response to the input coordinate value not being zero, encode the image index, the difference, and the growth direction according to a third encoding format to obtain continuous pixel coordinate encoding.

11. The apparatus according to claim 10, wherein, The state machine is also used to record the initial and final image calibration coordinates in each round of the first coordinate state; the second encoding format includes a first sub-format and a second sub-format, and the encoding sub-unit is further configured to: in response to the input coordinate value being non-zero and the number of image calibration coordinates of the image path reaching a preset number, encode the input coordinate value and image index according to the first sub-format to obtain a first initial coordinate code; in response to the input coordinate value being non-zero and the number of image calibration coordinates of the image path not reaching the preset number, calculate the difference between the final image calibration coordinate and the initial image calibration coordinate to obtain a continuous length; encode the image index, the continuous length, and the image calibration coordinate value according to the second sub-format to obtain a second initial coordinate code.

12. The apparatus according to any one of claims 7-11, wherein, The acquisition unit is further configured to: acquire videos from each fisheye camera in the multi-channel fisheye camera, perform boundary clipping on the videos, and decompose the clipped videos to obtain images from each channel of the multi-channel video; acquire the fisheye image coordinates of the images from the multi-channel cameras, correct the fisheye image coordinates to obtain the image calibration coordinates of each image in the multi-channel fisheye camera; and determine the fused image coordinates based on all image calibration coordinates.

13. A video fusion circuit, the circuit comprising: Multi-channel fisheye camera, field-programmable gate array circuit and synchronous dynamic random access memory; The synchronous dynamic random access memory is used to store multiple video streams captured by the multi-channel fisheye camera, as well as the image calibration coordinates and fused image coordinates of the multi-channel fisheye camera. The fused image coordinates are obtained by fusing all the image calibration coordinates. The field-programmable gate array circuit is used to obtain the weight parameters of the image pixels of each image in the multi-channel video based on the fused image coordinates, and store the weight parameters in the synchronous dynamic random access memory; Based on the image calibration coordinates, image pixels of each image in the multi-channel video are selected from the synchronous dynamic random access memory; Based on the weight parameters and pixel values ​​of each image, a fused image of all images in the multi-channel video is calculated.

14. The circuit according to claim 13, wherein, The field-programmable gate array circuit is also used to encode the image calibration coordinates to obtain coordinate information encoding, and store the coordinate information encoding in the synchronous dynamic random access memory; based on the coordinate information encoding, image pixels of each image in the multi-channel video are selected from the synchronous dynamic random access memory.

15. The circuit according to claim 14, wherein, The field-programmable gate array circuit includes: a first buffer and a second buffer, a comparator, a state machine, and an encoding circuit; The first buffer and the second buffer respectively store the calibration coordinates of two adjacent images; The comparator is used to compare any two adjacent image calibration coordinates in the image path to obtain the difference between the two image calibration coordinates and the direction of increase; The state machine performs encoding state transitions based on the input image calibration coordinate values ​​and the number of image calibration coordinates to obtain different encoding states; The encoding circuit encodes the image index, the image calibration coordinate values, the difference, and the growth direction according to the corresponding encoding format under different encoding states to obtain coordinate information encoding.

16. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

17. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images