A circuit and method for superimposing raindrop effect on digital video signal in real time

Abstract

The present invention provides a circuit and method for superimposing raindrop effects on digital video signals in real time, including a GPU video format conversion module, which converts a video stream into an RGB video stream and sends it to a raindrop sample cache module and a real-time frame picture replacement module at the same time; the raindrop sample cache module samples and shrinks each frame of RGB video to form a raindrop sample; the real-time frame picture replacement module replaces the pixel value of the pixel point in the current frame RGB video picture that is located in the raindrop area with a raindrop sample, and then sends it to a vehicle video format conversion module; the vehicle video format conversion module converts the RGB video stream superimposed with the raindrop effect into the format of the vehicle video stream, and injects the video into the automatic driving controller. The present invention can add the raindrop effect to the video stream in real time in a nearly zero-cost manner, thereby improving the realism and flexibility of the video injection into the automatic driving controller.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and in particular relates to a circuit and method for superimposing raindrop effects on digital video signals in real time. Background Art

[0002] The autonomous driving controller receives video signals transmitted from the vehicle's camera, analyzes and identifies the video signals, and then controls the vehicle's brakes, accelerator, steering and other mechanical interfaces to achieve autonomous driving control.

[0003] Therefore, when debugging or testing the autonomous driving controller, a key issue that needs to be addressed is how to input video signals under indoor conditions. Currently, a test method called "video injection" is usually used. Its basic principle is:

[0004] Play back or simulate the video signal of the vehicle camera on the computer, and output the video signal through the HDMI or displayport interface of the graphics card. Use a dedicated video conversion device to convert the HDMI or displayport digital video into FPD LINK or GMSL vehicle camera digital video. The device directly injects the vehicle camera digital video into the autonomous driving controller through the FAKRA interface. In this way, the autonomous driving controller can perform various tests and debugging under the condition of video input. Figure 1 shown.

[0005] However, this method has a trade-off between realism and flexibility. The details are as follows:

[0006] There are two ways to inject video sources into the autonomous driving controller. One is the video simulated by scene simulation software; the other is the video captured and played back through on-site video. The former has great flexibility, but it is essentially a simulated animation, which is quite different from the actual road video. Injection testing and debugging in this way have always been considered to be not realistic enough. The latter is played back through road video, and the realism can be guaranteed, but the flexibility is not enough, and the video cannot be changed at will. Generally speaking, road videos are all videos of "normal driving". However, the autonomous driving controller (actually the auxiliary driving controller at this stage) often needs to intervene in "crisis moments", such as car crashes, collisions, etc. Road videos in this case are difficult to obtain.

[0007] If you want to achieve the best state of realism and flexibility by purely improving the simulation realism of scene simulation software, it often involves the issue of simulation cost. At present, computer CG technology can indeed achieve realistic video simulation technology, but the premise is to consume a lot of computing resources and simulation time. This actually loses the meaning of flexibility itself.

[0008] In fact, in addition to car crashes and collisions, there are a large number of other scene requirements, such as rainy days. If you can add raindrop effects to actual road shooting videos at will without adding any time or financial costs, this will achieve the optimal configuration of compulsion and flexibility. Summary of the invention

[0009] The purpose of the present invention is to provide a circuit and method for superimposing raindrop effects on digital video signals in real time, which can add raindrop effects to video streams in real time at almost zero cost (including time and finance), thereby improving the realism and flexibility of video injection into an autonomous driving controller.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] The present invention provides a circuit for superimposing raindrop effects on digital video signals in real time, comprising a GPU video format conversion module, a raindrop sample buffer module, a real-time frame picture replacement module and a vehicle-mounted video format conversion module;

[0012] The GPU video format conversion module converts the video stream sent by the GPU graphics card into an RGB video stream, and sends the RGB video stream to the raindrop sample buffer module and the real-time frame picture replacement module at the same time;

[0013] The raindrop sample buffer module samples each received frame of RGB video and shrinks it to the size of a raindrop to form a raindrop sample;

[0014] The real-time frame picture replacement module replaces the pixel value of the pixel point in the current frame RGB video picture that is located in the raindrop area with the pixel value of the corresponding position in the raindrop sample, and then sends it to the vehicle video format conversion module;

[0015] The vehicle-mounted video format conversion module converts the received RGB video stream with superimposed raindrop effect into the format of the vehicle-mounted video stream, injects the video into the automatic driving controller, and realizes the real-time superimposition of raindrop effect on the digital video signal.

[0016] The raindrop sample buffer module includes a row coordinate counter, a column coordinate counter, two subtractors, four truncation modules, two merging modules, a ping-pong switching module, a two-choice module, two AND gates, a NOT gate, a delay module, and two dual-port random access memories DPRAMA and DPRAMB; the pixel clock PCLK, the line synchronization signal HS and the pixel effective signal DE in the RGB video stream signal output by the GPU video format conversion module are input into the row coordinate counter, and the pixel clock PCLK, the line synchronization signal HS, the pixel effective signal DE and the column synchronization signal VS in the RGB video stream signal are input into the column coordinate counter to obtain the row coordinate x_cor and column coordinate y_cor of the pixel point corresponding to the current RGB value in one frame, and the row coordinate and the column coordinate are respectively intercepted by the truncation module, and then subtracted from the maximum raindrop diameter in the subtractor, and then the subtraction results of the row coordinate and the column coordinate are merged in the merging module to obtain the A port write address addrA of the dual-port random access memories DPRAMA and DPRAMB; The sample read addresses ex_cor and ey_cor output by the real-time frame replacement module are intercepted by the truncation module respectively and then merged in the merging module to obtain the B port read address addrB of the dual-port random access memories DPRAMA and DPRAMB; the pixel point signal RGB[23:0] in the RGB video stream signal output by the GPU video format conversion module is delayed by the delay module and input into the dual-port random access memories DPRAMA and DPRAMB to form raindrop samples; the pixel clock PCLK and the column synchronization signal VS in the RGB video stream signal are input into the ping-pong switching module, and the output of the ping-pong switching module is logically ANDed with the pixel valid signal DE to form the A port write enable signal of the dual-port random access memories DPRAMA and DPRAMB, so that DPRAMA can only be written in odd frames, and DPRAMB can only be written in even frames; and the ping-pong switching module controls the two-choice module to output the raindrop samples of DPRAMB in odd frames and output the raindrop samples of DPRAMA in even frames.

[0017] The VerilogHDL language of the row coordinate counter and the column coordinate counter is:

[0018] module hvcount(

[0019] input clk,

[0020] input clear,

[0021] input cc,

[0022] output reg [15:0] count );

[0024] always@(posedge clk)

[0025] if (clear)

[0026] count <= 16'h0000;

[0027] else if (cc)

[0028] count <= count + 1;

[0029] endmodule

[0030] The corresponding connection relationship of the row coordinate counter is as follows:

[0031] external signal internal signal

[0032] PCLK <-> clk

[0033] HS <-> clear

[0034] DE <-> cc

[0035] x_cor <-> count

[0036] The corresponding connection relationship of the column coordinate counter is as follows:

[0037] external signal internal signal

[0038] PCLK <-> clk

[0039] VS <-> clear

[0040] HS <-> cc

[0041] y_cor <-> count .

[0042] The real-time frame picture replacement module includes n raindrop circle center distance calculation modules, a unique hot code decoding module, an n-to-one module, a two-to-one module and two delay modules; wherein the raindrop circle center distance calculation module is used to calculate the distance between the coordinate value of the current pixel point in the current frame RGB video picture output by the GPU video format conversion module and the coordinate value of the raindrop center point. When the distance is less than or equal to the raindrop radius, the pixel point needs to be replaced, otherwise it should not be replaced; at the same time, the raindrop circle center distance calculation module outputs the coordinate difference between the current pixel point and the raindrop diameter as a candidate address of the raindrop sample reading address to the n-to-one module; since there are multiple raindrops, and for any pixel point, it can only be in one raindrop at most In the example, the output result eflag of the n raindrop center distance calculation module is decoded by the one-hot code decoding module, and the specific raindrop where the current pixel is located can be obtained. The one-hot code decoding module has two outputs, one of which flows into the n-choose-one module to obtain the sample reading address for the raindrop sample cache module, and the other output enters the two-choose-one module together with the raindrop sample output by the raindrop sample cache module and the current video stream RGB delayed by the delay circuit. The two-choose-one module is used to select whether to replace the pixel with the raindrop sample. The output value of the two-choose-one module and the RGB video stream signal delayed by the delay circuit together constitute an RGB video stream with a superimposed raindrop effect, which flows into the on-board video format conversion module.

[0043] The raindrop center distance calculation module includes two subtraction modules, two multiplication modules, an addition module and a comparison module. First, the row coordinates and column coordinates of the current pixel point are respectively subtracted from the row coordinates and column coordinates of the raindrop center point in the two subtraction modules, and then the two subtraction results are respectively entered into the two multiplication modules for square calculation, and the two multiplication results are then entered into the addition module for addition to obtain the distance between the current pixel point and the raindrop center point. The distance is numerically compared with the square value of the raindrop radius in the comparison module. If the comparison result is 1, it indicates that the current pixel point is outside the raindrop, otherwise, it indicates that the current pixel point is inside the raindrop.

[0044] The comparison module, under the condition of clk rising edge, has two inputs a and b. When a>b, the output is y=1; otherwise, y=0. Its VerilogHDL language is:

[0045] module compare(

[0046] input clk,

[0047] input [15:0] a,

[0048] input [15:0] b,

[0049] output reg y );

[0051] always@(posedge clk)

[0052] if (a>b)

[0053] y <= 1'b1;

[0054] else

[0055] y <= 1'b0;

[0056] endmodule .

[0057] The one-hot code decoding module is related to the specific raindrop value n. Taking n=4 as an example, its Verilog HDL language is:

[0058] module decode(

[0059] input clk,

[0060] input [3:0] eflag,

[0061] output reg [1:0] y,

[0062] output reg dot );

[0064] always@(posedge clk)

[0065] if (eflag[0])

[0066] y <= 2'b00;

[0067] else if (eflag[1])

[0068] y <= 2'b01;

[0069] else if (eflag[2])

[0070] y <= 2'b10;

[0071] else if (eflag[3])

[0072] y <= 2'b11;

[0073] always@(posedge clk)

[0074] dot <= |eflag;

[0075] endmodule .

[0076] The VerilogHDL language of the two-choice module in the real-time frame picture replacement module is:

[0077] module mux2to1(

[0078] input clk,

[0079] input [23:0] a,

[0080] input [23:0] b,

[0081] input sel,

[0082] output reg [23:0] y );

[0084] always@(posedge clk)

[0085] if (sel)

[0086] y <= b;

[0087] else

[0088] y <= a;

[0089] endmodule

[0090] When the selection input sel is 1, the output y is the value of input b; otherwise, the output y is the value of a;

[0091] The corresponding connection relationship of the two-choose-one module is as follows:

[0092] external signal internal signal

[0093] PCLK <-> clk

[0094] Raindrop sample RGB <-> a

[0095] Current video stream RGB <-> b

[0096] Dot <-> sel

[0097] eRGB <-> y .

[0098] The n-choose-1 module is specifically related to n. Taking n=4 as an example, its VerilogHDL language is:

[0099] module mux4to1(

[0100] input clk,

[0101] input [31:0] din0,

[0102] input [31:0] din1,

[0103] input [31:0] din2,

[0104] input [31:0] din3,

[0105] input [1:0] sel,

[0106] output reg [31:0] dout );

[0108] always@(posedge clk)

[0109] case (sel)

[0110] 2'b00: dout<= din0;

[0111] 2'b01: dout<= din1;

[0112] 2'b10: dout<= din2;

[0113] 2'b11: dout<= din3;

[0114] endcase

[0115] endmodule .

[0116] A method for superimposing raindrop effects using the circuit for superimposing raindrop effects on digital video signals in real time is characterized by comprising the following steps:

[0117] 1) The video stream sent by the GPU graphics card is converted into an RGB video stream by the GPU video format conversion module, and is sent to the raindrop sample buffer module and the real-time frame replacement module at the same time;

[0118] 2) The real-time frame replacement module caches the current frame RGB video image;

[0119] 3) The raindrop sample cache module shrinks the current frame RGB video image according to the size of the raindrop and caches the shrink image; then, according to the center position of the raindrop i, a circular area is drawn with the center position as the center and the raindrop size as the diameter to form a raindrop sample; where i=1…n, n is the total number of raindrops;

[0120] 4) The real-time frame replacement module calls the cached last frame of RGB video image and replaces the circular area image of raindrop i with the miniature image;

[0121] 5) Determine whether raindrop i is the last raindrop. If not, update raindrop i to raindrop i+1, and then return to step 3). If so, the current frame RGB video image has completed the superposition of the raindrop effect, and the superimposed RGB video image is sent to the vehicle video format conversion module for vehicle video stream format conversion. At the same time, the raindrop sample cache module and the real-time frame picture replacement module start to perform the above-mentioned raindrop effect superposition processing on the next frame RGB video image.

[0122] Compared with the prior art, the beneficial effects of the present invention are:

[0123] The present invention provides a circuit and method for superimposing raindrop effects on digital video signals in real time, which can superimpose raindrop effects in real time while uninterrupted video streaming. The realism and flexibility of video injection into the automatic driving controller can be improved. When video injection is implemented using FPGA devices, the implementation of this technology does not require any cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1 It is a structural diagram of an existing video injection system;

[0125] Figure 2 Schematic diagram of the imaging relationship between water droplets and the lens;

[0126] Figure 3 The overall flow chart for real-time raindrop effect overlay;

[0127] Figure 4 This is the effect picture of superimposed water drops;

[0128] Figure 5 Circuit diagram for real-time raindrop effect overlay;

[0129] Figure 6 This is the principle block diagram of the raindrop sample buffer module circuit;

[0130] Figure 7 It is the principle block diagram of the real-time frame picture replacement module circuit;

[0131] Figure 8 This is the principle block diagram of the raindrop center distance calculation module. DETAILED DESCRIPTION

[0132] The present invention is further described below in conjunction with the accompanying drawings. The following examples are only used to describe the present invention and are not used to limit the scope of use of the present invention. Various equivalent modifications of the present invention by engineers and technicians in various fields are included in the scope of the rights required by the present invention.

[0133] The principle of real-time raindrop effect superposition technology

[0134] The real-time raindrop effect overlay technology is achieved by modifying each frame in the video stream. This technology uses a circuit structure to replace some pixels in each frame of the video stream through stream processing. The replaced pixels are the water drop effect to be created.

[0135] For actual car cameras, only those water drops that fall in front of the camera lens will affect the actual video stream. In order to protect the lens, generally speaking, a flat glass cover is placed in front of the car camera lens for protection. Therefore, the place where water drops can actually adhere is this glass cover. Water drops and the lens present a special imaging relationship, as shown below: Figure 2 shown.

[0136] When a water droplet adheres to the glass cover in front of the lens, it is affected by the surface tension of the water droplet to form an approximate convex lens. In this way, the lens inside the camera and the convex lens formed by the water droplet are combined into a new lens group. The imaging principle of this new lens group is as follows Figure 2 shown.

[0137] Suppose there are two physical points A and B. For the camera lens, points A and B will form two points A1 and B1 on the photosensitive surface of the camera. After adding the water drop convex lens, the incident light from points A and B will be refracted by the water drop convex lens, and a smaller virtual image will be formed between the water drop convex lens and the lens. The light of this virtual image will be refracted by the lens again to form two points A2 and B2 on the photosensitive surface.

[0138] from Figure 2 It can be seen that the positions of A1, B1 and A2, B2 are reversed, and the distance between A1 and B1 is greater than the distance between A2 and B2.

[0139] From this, we can see that when water droplets adhere to the camera, the impact on the final image can be roughly considered as follows:

[0140] In a certain area of ​​the video, an inverted picture is superimposed. The picture is a miniature of the current frame. The size and position of this area are related to the size and position of the raindrops.

[0141] The so-called raindrop effect is the effect of several or more small raindrops with random positions. Therefore, in actual simulation, the shape of the raindrop can be approximated as a circle, the position of the raindrop can be set to a random value, and the range of the miniature can be approximated to the entire single frame.

[0142] To sum up, the overall implementation process of real-time raindrop effect superposition technology is as follows: Figure 3 shown.

[0143] according to Figure 3 The implementation process shown in the figure shows the superimposed water drop effect. Figure 4 shown.

[0144] Circuit design for implementing real-time raindrop effect superposition technology

[0145] The present invention provides a circuit for superimposing raindrop effects on digital video signals in real time (raindrop effect superimposition circuit), and its structure is as follows: Figure 5 shown.

[0146] The circuit is generally divided into four modules: GPU video format conversion module, real-time frame replacement module, raindrop sample cache module and vehicle-mounted video format conversion module.

[0147] The main functions of each module are:

[0148] The GPU video format conversion module is mainly responsible for converting the video stream sent by the GPU graphics card into an RGB video stream for subsequent processing.

[0149] The real-time frame replacement module is mainly responsible for replacing certain areas in the current frame with raindrop samples.

[0150] The raindrop sample cache module is mainly responsible for sampling and miniaturizing the RGB video frame into raindrop samples of a set size. The sample is updated every frame. It has a ping-pong structure internally. The miniature of the previous frame is stored in storage area A, and the sample is read from area A; the miniature of the current frame is stored in storage area B. In the next frame, the sample is read from area B, and the miniature of the current frame is stored in storage area A, and this cycle repeats.

[0151] The on-board video format conversion module is mainly responsible for converting the RGB video stream superimposed with the raindrop effect into the format of the on-board video stream, which can directly inject the video into the autonomous driving controller.

[0152] The GPU graphics card sends a video stream in HDMI or Displayport format to the GPU video format conversion module in the raindrop effect superposition circuit. After the GPU video format conversion module converts the video stream into an easy-to-process RGB video stream, it sends it to the raindrop sample cache module and the real-time frame picture replacement module at the same time. The raindrop sample cache module samples each frame of RGB video and shrinks the frame picture to the size of raindrops to form raindrop samples. The raindrop sample cache module internally forms a ping-pong cache structure to ensure that each raindrop sample is complete. The real-time frame picture replacement module determines the position information of the pixel point of the video stream according to the count value of the current video stream. When the position information is located in the raindrop area, the pixel value of the point is replaced with the pixel value of the corresponding position in the raindrop sample, and then continues to output. In this way, the RGB video stream output by the real-time frame picture replacement module is the RGB video stream with the raindrop effect superimposed. The RGB video stream is sent to the vehicle-mounted video format conversion module, which converts the RGB video stream into a vehicle-mounted camera video format such as GMSL or FPD Link, and finally injects the vehicle-mounted video stream into the autonomous driving controller.

[0153] The GPU video format conversion module and the vehicle-mounted video format conversion module can be implemented by methods well known in the industry and already disclosed. Therefore, the core focus of the present invention is on the circuit design of the raindrop sample buffer module and the real-time frame picture replacement module.

[0154] The implementation methods of the GPU video format conversion module and the vehicle-mounted video format conversion module are as follows:

[0155] 1. Implementation of GPU video format conversion module: including but not limited to the following examples

[0156] a) The GPU video format conversion module can be implemented by the ADV7611 chip of Analog Devices. This chip can convert HDMI format input into RGB video stream output.

[0157] b) The GPU video format conversion module can be implemented in Xilinx's FPGA through Xilinx's Displayport RX IP.

[0158] Implementation of the vehicle video format conversion module: including but not limited to the following examples:

[0159] a) It can be realized by Maxim's MAX96705 chip. This chip can convert RGB video stream (or YUV video stream) input into GMSL on-board video stream output.

[0160] b) It can be realized by TI's DS90UB953 chip. This chip can convert RGB video stream (or YUV video stream) input into FPD Link vehicle-mounted video stream output.

[0161] For the convenience of subsequent discussion, the RGB video stream format is briefly described below. This format has long been public and well known in the industry.

[0162] The RGB video stream includes the following signals:

[0163] Pixel clock PCLK, the rising edge is one pixel clock.

[0164] b. Frame synchronization signal - VS, the rising edge indicates the start of the frame, and the high level only lasts for one clock.

[0165] c. Line synchronization signal - HS, the rising edge indicates the start of a line, and the high level only lasts for one clock.

[0166] d. Pixel valid signal - DE, high level and PCLK rising edge, indicating that the RGB pixel signal at this moment is a valid pixel.

[0167] e. Pixel signal - RGB[23:0], RGB[7:0] is the blue pixel value, RGB[15:8] is the green pixel value, and RGB[23:16] is the red pixel value.

[0168] A set of RGB video streams consists of 28 signals, which use the rising edge of PCLK as the clock beat to transmit the video stream.

[0169] When describing the specific circuit implementation in the following, the synthesizable VerilogHDL language description will be used. This language is a standardized hardware description language well known in the industry. The actual circuit module can be formed by using the synthesizer and automatic layout and routing tools that are open and well known in the industry. These synthesizers and automatic layout and routing tools include but are not limited to:

[0170] Xilinx Vivado

[0171] Cadance RTL Compiler

[0172] Synopsys Design Compile

[0173] Synopsys Astro

[0174] The specific implementation methods of the circuit described below include but are not limited to the following four:

[0175] 1. Use programmable logic chips such as FPGA to burn the circuit into the chip to realize the circuit function.

[0176] 2. According to the ASIC design and development process, the circuit is designed into an integrated circuit and taped out for production.

[0177] 3. Use basic logic components to build circuits on PCB circuit boards.

[0178] 4. Combining the above three methods.

[0179] The above methods are well known in the industry and need no further explanation of circuit implementation.

[0180] The circuit designs of the raindrop sample buffer module and the real-time frame picture replacement module of the present invention are described in detail below.

[0181] 1. Raindrop sample cache module

[0182] The principle block diagram of the raindrop sample buffer module circuit is as follows: Figure 6 shown.

[0183] The input of this circuit is the video stream signal, including PCLK, VS, HS, DE and the current video stream RGB[23:0]; there are also sample read addresses ex_cor and ey_cor.

[0184] The output signals of this circuit are raindrop samples RGB[23:0], row and column coordinates x_cor and y_cor.

[0185] The circuit consists of a row coordinate counter, a column coordinate counter, two subtractors, four truncation modules, two merging modules, a ping-pong switching module, a two-choice module, two AND gates, a NOT gate, a delay module and two dual-port random access memories (DPRAM).

[0186] The implementation circuits of the row coordinate counter and the column coordinate counter are exactly the same. The VerilogHDL language description is:

[0187] module hvcount(

[0188] input clk,

[0189] input clear,

[0190] input cc,

[0191] output reg [15:0] count );

[0193] always@(posedge clk)

[0194] if (clear)

[0195] count <= 16'h0000;

[0196] else if (cc)

[0197] count <= count + 1;

[0198] endmodule

[0199] The corresponding connection relationship of the row coordinate counter is as follows:

[0200] external signal internal signal

[0201] PCLK <-> clk

[0202] HS <-> clear

[0203] DE <-> cc

[0204] x_cor <-> count

[0205] When HS is high, the count value of the row coordinate counter is cleared. When DE is high and HS is low, the count value increases by one at each rising edge of the PCLK clock. In fact, it indicates the row coordinate of the pixel point corresponding to the current RGB value in a frame. Figure 6 It is indicated as x_cor in .

[0206] The corresponding connection relationship of the column coordinate counter is as follows:

[0207] external signal internal signal

[0208] PCLK <-> clk

[0209] VS <-> clear

[0210] HS <-> cc

[0211] y_cor <-> count

[0212] When VS is high, the count value of the column coordinate counter is cleared. When HS is high and VS is low, the count value increases by one at each rising edge of the PCLK clock. In fact, it indicates the column coordinates of the pixel corresponding to the current RGB value in a frame. Figure 6 It is indicated as y_cor in FIG.

[0213] After calculating the row and column coordinate values ​​of the current pixel point, the value is sent to the real-time frame image replacement module.

[0214] Considering that the size of raindrops will not reach the entire picture, there is no need to cache the pixel information of the entire frame. In order to save cache space, for a frame, the maximum raindrop diameter can be sampled. At the same time, in order to simplify the circuit, the cache size can be set to a power of 2. In this way, the sampling process can be directly implemented by truncating the row and column coordinates of the current pixel point. For example: for a frame of 1920×1080, the raindrop cache size is set to 16×16. The last 7 bits of x_cor can be truncated, leaving only x_cor[11:8]; the last 6 bits and the highest bit of y_cor can be truncated, leaving only y_cor[10:7].

[0215] According to the lens principle described above, the image formed by the raindrops and the image formed by the lens are upside down and left to right. Therefore, when caching, the pixel sample represented by address 0 should actually appear at the last address. Therefore, after truncating x_cor and y_cor, a subtraction operation is performed on x_cor and y_cor using the largest raindrop diameter. Finally, the calculated row and column coordinate results are bit-wise merged into a binary number, with the column coordinate placed in the high bit and the row coordinate placed in the low bit, thus obtaining the write address addrA of port A of DPRAM.

[0216] The sample read addresses ex_cor and ey_cor are calculated by the real-time frame image replacement module. The specific calculation method is described in the subsequent circuit design description of the real-time frame image replacement module. The sample read address is truncated by the truncation method described above to form the DPRAM B port read address addrB.

[0217] Two DPRAMs are designed in the raindrop sample cache module circuit, namely DPRAMA and DPRAMB. DPRAM is a dual-port RAM, where address port A is write-only and address port B is read-only. DPRAM is a well-known technology in the industry, with a large amount of information and is very mature, so it will not be described here.

[0218] At the same time, the raindrop sample cache module circuit designs a ping-pong switching module to control the read and write operations of the two DPRAMs. The ping-pong switching module inverts the output when each VS signal is high. In this way, the output of the odd frame is 1, and the output of the even frame is 0. Using this mark, DPRAMA can be in the write state only in odd frames, and DPRAMB can be in the write state only in even frames. Then use a two-choice module to output the raindrop samples of DPRAMB in odd frames; and output the raindrop samples of DPRAMA in even frames. Thus, the ping-pong operation is completed.

[0219] The circuit of the delay module uses a shift register to delay the output signal by x clocks relative to the input signal. Xilinx provides the IP of the delay module.

[0220] 2. Real-time frame replacement module

[0221] The principle block diagram of the real-time frame picture replacement module circuit of the present invention is as follows Figure 7 shown.

[0222] The circuit consists of n raindrop center distance calculation modules, a one-hot code decoding module, an n-choose-one module, a two-choose-one module, and two delay modules.

[0223] In a frame image, the pixel points that need to be replaced with raindrop samples are the area of ​​a circle with the raindrop as the center and the raindrop size as the radius. This area can be calculated by the distance between the pixel coordinate value and the raindrop center coordinate. When this distance is less than or equal to the raindrop radius, it indicates that the pixel point needs to be replaced; when this distance is greater than the raindrop radius, it indicates that the pixel point does not need to be replaced.

[0224] According to the above principle, a raindrop center distance calculation module is designed, such as Figure 8 shown.

[0225] The raindrop center distance calculation module consists of two subtraction modules, two multiplication modules, an addition module and a comparison module. This module subtracts the row and column coordinates of the current pixel point from the row and column coordinates of the raindrop center point, and then calculates the square sum. This realizes the distance calculation between the current pixel point and the raindrop center point. The result is numerically compared with the square value of the raindrop radius using the comparison module. If the comparison result is 1, it means that the current pixel point is outside the raindrop; otherwise, it means that the current pixel point is inside the raindrop.

[0226] At the same time, the coordinate difference (xx_cor, yy_cor) between the current pixel point and the raindrop diameter calculated by the module will be output to the external circuit as a candidate address for the raindrop sample reading address.

[0227] The subtraction module, multiplication module and addition module are well known in the industry and there is a large amount of public information, so they will not be described in detail.

[0228] Comparison module: Under the rising edge condition of clk, two inputs a and b, when a>b, the output y=1; otherwise, y=0;

[0229] Its VerilogHDL language description is as follows:

[0230] module compare(

[0231] input clk,

[0232] input [15:0] a,

[0233] input [15:0] b,

[0234] output reg y );

[0236] always@(posedge clk)

[0237] if (a>b)

[0238] y <= 1'b1;

[0239] else

[0240] y <= 1'b0;

[0241] endmodule

[0242] Since there are multiple raindrops in an image, n raindrop center distance calculation modules are designed to correspond to n raindrops. In order to simplify the circuit design, the following settings are made during the raindrop effect superposition process:

[0243] For any pixel, it can only be in one raindrop at most.

[0244] Therefore, in order to select the specific raindrop where the current pixel is located, a one-hot code decoding module is designed to decode the output result eflag of the n raindrop center distance calculation module to obtain the corresponding sequence.

[0245] The design of the one-hot code decoding module is related to the specific value n. The following takes n=4 as an example (4 raindrops) to describe the Verilog HDL language.

[0246] module decode(

[0247] input clk,

[0248] input [3:0] eflag,

[0249] output reg [1:0] y,

[0250] output reg dot );

[0252] always@(posedge clk)

[0253] if (eflag[0])

[0254] y <= 2'b00;

[0255] else if (eflag[1])

[0256] y <= 2'b01;

[0257] else if (eflag[2])

[0258] y <= 2'b10;

[0259] else if (eflag[3])

[0260] y <= 2'b11;

[0261] always@(posedge clk)

[0262] dot <= |eflag;

[0263] endmodule

[0264] The one-hot decoding module has two outputs. Output y is the one-hot decoding result, and output dot is high if the input eflag sequence contains 1, otherwise it is low.

[0265] A two-choice module is designed in the real-time frame replacement module circuit to select whether to replace the pixel with a raindrop sample. The VerilogHDL language description of this module is as follows:

[0266] module mux2to1(

[0267] input clk,

[0268] input [23:0] a,

[0269] input [23:0] b,

[0270] input sel,

[0271] output reg [23:0] y );

[0273] always@(posedge clk)

[0274] if (sel)

[0275] y <= b;

[0276] else

[0277] y <= a;

[0278] endmodule

[0279] When the selection input sel is 1, the output y is the value of the input b; otherwise, the output y is the value of a.

[0280] The corresponding connection relationship of the two-choose-one module is as follows:

[0281] external signal internal signal

[0282] PCLK <-> clk

[0283] Raindrop sample RGB <-> a

[0284] Current video stream RGB <-> b

[0285] Dot <-> sel

[0286] eRGB <-> y

[0287] Since the distance calculation between the pixel and the center of the raindrop involves addition, subtraction, and multiplication, a fixed number of clocks are required. The specific clock value is determined by the clock frequency and chip process. Therefore, the final comparator output result will be delayed by several clocks compared to the current video stream. In order to ensure synchronization, a delay module is added to the current video stream before the two-choice module, and the four signals PCLK, VS, HS, and DE are also delayed to make up for the delayed clock number. Finally, the video stream after superimposing the raindrops is formed:

[0288] ePCLK

[0289] eVS

[0290] eHS

[0291] eDE

[0292] eRGB[23:0]

[0293] When a pixel is determined to be within the raindrop circle, it is the pixel that needs to be replaced with the raindrop sample. Then, it is necessary to calculate the coordinate value of the pixel in the corresponding raindrop sample based on the coordinate value of the pixel. The calculation method is to subtract the row and column coordinates of the current pixel from the row and column coordinates of the raindrop center point, and address the pixel value in the raindrop sample cache area according to the new coordinates obtained. The above subtraction is consistent with the subtraction of calculating the center distance, so it can be merged in hardware. Since there are multiple raindrop center distance calculation modules, it is necessary to select the raindrop that is determined to be included in the current pixel based on the result of the one-hot code decoding module. Therefore, an n-to-1 module is designed. The specific design of this module is related to n. Here, n=4 is taken as an example to describe the VerilogHDL design:

[0294] module mux4to1(

[0295] input clk,

[0296] input [31:0] din0,

[0297] input [31:0] din1,

[0298] input [31:0] din2,

[0299] input [31:0] din3,

[0300] input [1:0] sel,

[0301] output reg [31:0] dout );

[0303] always@(posedge clk)

[0304] case (sel)

[0305] 2'b00: dout<= din0;

[0306] 2'b01: dout<= din1;

[0307] 2'b10: dout<= din2;

[0308] 2'b11: dout<= din3;

[0309] endcase

[0310] endmodule

[0311] The 32-bit output dout of the above circuit is split by bit, the lower 16 bits are ex_cor[15:0], and the upper 16 bits are ey_cor[15:0], which forms the read address signal for the raindrop sample cache module.

[0312] A method for superimposing raindrop effects by using the circuit for superimposing raindrop effects on digital video signals in real time according to the present invention is as follows: Figure 3 As shown, the following steps are included:

[0313] 1) The video stream sent by the GPU graphics card is converted into an RGB video stream by the GPU video format conversion module, and is sent to the raindrop sample buffer module and the real-time frame replacement module at the same time;

[0314] 2) The real-time frame replacement module caches the current frame RGB video image;

[0315] 3) The raindrop sample cache module shrinks the current frame RGB video image according to the size of the raindrop and caches the shrink image; then, according to the center position of the raindrop i, a circular area is drawn with the center position as the center and the raindrop size as the diameter to form a raindrop sample; where i=1…n, n is the total number of raindrops;

[0316] 4) The real-time frame replacement module calls the cached last frame of RGB video image and replaces the circular area image of raindrop i with the miniature image;

[0317] 5) Determine whether raindrop i is the last raindrop. If not, update raindrop i to raindrop i+1, and then return to step 3). If so, the current frame RGB video image has completed the superposition of the raindrop effect, and the superimposed RGB video image is sent to the vehicle video format conversion module for vehicle video stream format conversion. At the same time, the raindrop sample cache module and the real-time frame picture replacement module start to perform the above-mentioned raindrop effect superposition processing on the next frame RGB video image.

[0318] The present invention provides a circuit and method for superimposing raindrop effects on digital video signals in real time, which can add raindrop effects to the video stream in real time while uninterrupted video streaming, thereby improving the realism and flexibility of video injection into the automatic driving controller.

[0319] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural transformation made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A circuit for superimposing raindrop effects on digital video signals in real time. Features: It includes GPU video format conversion module, raindrop sample cache module, real-time frame replacement module and vehicle video format conversion module; The GPU video format conversion module converts the video stream sent by the GPU graphics card into an RGB video stream, and sends the RGB video stream to the raindrop sample buffer module and the real-time frame picture replacement module at the same time; The raindrop sample buffer module samples each received frame of RGB video and shrinks it to the size of a raindrop to form a raindrop sample; The real-time frame picture replacement module replaces the pixel value of the pixel point in the current frame RGB video picture that is located in the raindrop area with the pixel value of the corresponding position in the raindrop sample, and then sends it to the vehicle video format conversion module; The vehicle-mounted video format conversion module converts the received RGB video stream with the superimposed raindrop effect into the format of the vehicle-mounted video stream, and injects the video into the automatic driving controller, so as to realize the real-time superimposition of the raindrop effect on the digital video signal; The raindrop sample buffer module includes a row coordinate counter, a column coordinate counter, two subtractors, four truncation modules, two merging modules, a ping-pong switching module, a two-choice module, two AND gates, a NOT gate, a delay module, and two dual-port random access memories DPRAMA and DPRAMB; the pixel clock PCLK, the line synchronization signal HS and the pixel effective signal DE in the RGB video stream signal output by the GPU video format conversion module are input into the row coordinate counter, and the pixel clock PCLK, the line synchronization signal HS, the pixel effective signal DE and the column synchronization signal VS in the RGB video stream signal are input into the column coordinate counter to obtain the row coordinate x_cor and column coordinate y_cor of the pixel point corresponding to the current RGB value in one frame, and the row coordinate and the column coordinate are respectively intercepted by the truncation module, and then subtracted from the maximum raindrop diameter in the subtractor, and then the subtraction results of the row coordinate and the column coordinate are merged in the merging module to obtain the A port write address addrA of the dual-port random access memories DPRAMA and DPRAMB; The sample read addresses ex_cor and ey_cor output by the real-time frame replacement module are intercepted by the truncation module respectively and then merged in the merging module to obtain the B port read address addrB of the dual-port random access memories DPRAMA and DPRAMB; the pixel point signal RGB[23:0] in the RGB video stream signal output by the GPU video format conversion module is delayed by the delay module and input into the dual-port random access memories DPRAMA and DPRAMB to form raindrop samples; the pixel clock PCLK and the column synchronization signal VS in the RGB video stream signal are input into the ping-pong switching module, and the output of the ping-pong switching module is logically ANDed with the pixel valid signal DE to form the A port write enable signal of the dual-port random access memories DPRAMA and DPRAMB, so that DPRAMA can only be written in odd frames, and DPRAMB can only be written in even frames; and the ping-pong switching module controls the two-choice module to output the raindrop samples of DPRAMB in odd frames and output the raindrop samples of DPRAMA in even frames.

2. The circuit for superimposing raindrop effect on digital video signal in real time according to claim 1, Features: The VerilogHDL language of the row coordinate counter and the column coordinate counter is: module hvcount( input clk, input clear, input cc, output reg [15:0] count ); always@(posedge clk) if (clear) count <= 16'h0000; else if (cc) count <= count + 1; endmodule The corresponding connection relationship of the row coordinate counter is as follows: external signal internal signal PCLK <-> clk HS <-> clear DE <-> cc x_cor <-> count The corresponding connection relationship of the column coordinate counter is as follows: external signal internal signal PCLK <-> clk VS <-> clear HS <-> cc y_cor <-> count .

3. The circuit for superimposing raindrop effect on digital video signal in real time according to claim 1 or 2, Features: The real-time frame picture replacement module includes n raindrop circle center distance calculation modules, a unique hot code decoding module, an n-to-one module, a two-to-one module and two delay modules; wherein the raindrop circle center distance calculation module is used to calculate the distance between the coordinate value of the current pixel point in the current frame RGB video picture output by the GPU video format conversion module and the coordinate value of the raindrop center point. When the distance is less than or equal to the raindrop radius, the pixel point needs to be replaced, otherwise it should not be replaced; at the same time, the raindrop circle center distance calculation module outputs the coordinate difference between the current pixel point and the raindrop diameter as a candidate address of the raindrop sample reading address to the n-to-one module; since there are multiple raindrops, and for any pixel point, it can only be in one raindrop at most In the example, the output result eflag of the n raindrop center distance calculation module is decoded by the one-hot code decoding module, and the specific raindrop where the current pixel is located can be obtained. The one-hot code decoding module has two outputs, one of which flows into the n-choose-one module to obtain the sample reading address for the raindrop sample cache module, and the other output enters the two-choose-one module together with the raindrop sample output by the raindrop sample cache module and the current video stream RGB delayed by the delay circuit. The two-choose-one module is used to select whether to replace the pixel with the raindrop sample. The output value of the two-choose-one module and the RGB video stream signal delayed by the delay circuit together constitute an RGB video stream with a superimposed raindrop effect, which flows into the on-board video format conversion module.

4. The circuit for superimposing raindrop effect on digital video signal in real time according to claim 3, Features: The raindrop center distance calculation module includes two subtraction modules, two multiplication modules, an addition module and a comparison module. First, the row coordinates and column coordinates of the current pixel point are respectively subtracted from the row coordinates and column coordinates of the raindrop center point in the two subtraction modules, and then the two subtraction results are respectively entered into the two multiplication modules for square calculation, and the two multiplication results are then entered into the addition module for addition to obtain the distance between the current pixel point and the raindrop center point. The distance is numerically compared with the square value of the raindrop radius in the comparison module. If the comparison result is 1, it indicates that the current pixel point is outside the raindrop, otherwise, it indicates that the current pixel point is inside the raindrop.

5. The circuit for superimposing raindrop effect on digital video signal in real time according to claim 4, Features: The comparison module, under the condition of clk rising edge, has two inputs a and b. When a>b, the output is y=1; otherwise, y=0. Its VerilogHDL language is: module compare( input clk, input [15:0] a, input [15:0] b, output reg y ); always@(posedge clk) if (a>b) y <= 1'b1; else y <= 1'b0; endmodule .

6. The circuit for superimposing raindrop effect on digital video signal in real time according to claim 3, Features: The one-hot code decoding module is related to the specific raindrop value n, and its Verilog HDL language is: module decode( input clk, input [3:0] eflag, output reg [1:0] y, output reg dot ); always@(posedge clk) if (eflag[0]) y <= 2'b00; else if (eflag[1]) y <= 2'b01; else if (eflag[2]) y <= 2'b10; else if (eflag[3]) y <= 2'b11; always@(posedge clk) dot <= |eflag; endmodule .

7. The circuit for superimposing raindrop effect on digital video signal in real time according to claim 3, Features: The VerilogHDL language of the two-choice module in the real-time frame picture replacement module is: module mux2to1( input clk, input [23:0] a, input [23:0] b, input sel, output reg [23:0] y ); always@(posedge clk) if (sel) y <= b; else y <= a; endmodule When the selection input sel is 1, the output y is the value of input b; otherwise, the output y is the value of a; The corresponding connection relationship of the two-choose-one module is as follows: external signal internal signal PCLK <-> clk Raindrop sample RGB <-> a Current video stream RGB <-> b Dot <-> sel eRGB <-> y .

8. The circuit for superimposing raindrop effect on digital video signal in real time according to claim 3, Features: The n-choose-one module is specifically related to n, and its VerilogHDL language is: module mux4to1( input clk, input [31:0] din0, input [31:0] din1, input [31:0] din2, input [31:0] din3, input [1:0] sel, output reg [31:0] dout ); always@(posedge clk) case (sel) 2'b00: dout<= din0; 2'b01: dout<= din1; 2'b10: dout<= din2; 2'b11: dout<= din3; endcase endmodule .

9. A method for superimposing raindrop effects using the circuit for superimposing raindrop effects on digital video signals in real time according to claim 1, It is characterized in that The following steps are involved: 1) The video stream sent by the GPU graphics card is converted into an RGB video stream by the GPU video format conversion module, and is sent to the raindrop sample buffer module and the real-time frame replacement module at the same time; 2) The real-time frame replacement module caches the current frame RGB video image; 3) The raindrop sample cache module shrinks the current frame RGB video image according to the size of the raindrop and caches the shrink image; then, according to the center position of the raindrop i, a circular area is drawn with the center position as the center and the raindrop size as the diameter to form a raindrop sample; where i=1…n, n is the total number of raindrops; 4) The real-time frame replacement module calls the cached last frame of RGB video image and replaces the circular area image of raindrop i with the miniature image; 5) Determine whether raindrop i is the last raindrop. If not, update raindrop i to raindrop i+1, and then return to step 3). If so, the current frame RGB video image has completed the superposition of the raindrop effect, and the superimposed RGB video image is sent to the vehicle video format conversion module for vehicle video stream format conversion. At the same time, the raindrop sample cache module and the real-time frame picture replacement module start to perform the above-mentioned raindrop effect superposition processing on the next frame RGB video image.

Images