Video Processing Method

Abstract

To appropriately record color video.SOLUTION: A video processing method for processing color video captured by imaging means which captures a traffic situation around a vehicle, includes: a retaining step of retaining, in color, a first video image part of the video in a first range that is smaller than a range captured by the imaging means; and a conversion step of converting a second video image part containing at least a second range other than the first range of the video into monochrome.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to the technical field of video processing methods. [Background technology]

[0002] As one example of this type of method, a method has been proposed in which, based on the brightness of a portion of the image frame of a video signal output by an on-board camera, including the interior of the vehicle, color images are recorded when the surroundings are bright, such as during the day, and monochrome images are recorded when the surroundings are dark, such as at night (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6317914 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, either color or monochrome is selected depending on the brightness of the surroundings of the vehicle. However, when the surroundings of the vehicle are dark, selecting monochrome images may not be appropriate. A technology has also been proposed that uses a computational model constructed by machine learning to colorize monochrome images. However, colorization may not reproduce the original colors.

[0005] The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing method that can appropriately record color images. [Means for solving the problem]

[0006] A video processing method according to one aspect of the present invention includes: An image processing method for processing color images captured by an imaging means for capturing images of traffic conditions around a vehicle, the method comprising: a retaining step of retaining, in color, a first image portion of the image having a first range smaller than the range captured by the imaging means; and a converting step of converting, to monochrome, a second image portion of the image including at least a second range other than the first range. a transmitting step of transmitting the first video portion held in color and the second video portion converted to monochrome to an external device different from the vehicle; and a combining step of colorizing the second video portion converted to monochrome by the external device and combining the colorized second video portion with the first video portion held in color. It includes the following. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing the configuration of a vehicle and a receiving device according to an embodiment; [Figure 2] FIG. 1 is a diagram illustrating a concept of video processing according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating a concept of video processing according to an embodiment. [Figure 4] 4 is a flowchart illustrating an operation of a video processing unit according to the embodiment. [Figure 5] 4 is a flowchart illustrating an operation of a video processing unit according to the embodiment. [Figure 6] 10 is a flowchart illustrating an operation of a receiving side device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] An embodiment of the image processing method will be described with reference to Figs. 1 to 5. In Fig. 1, a vehicle 1 includes an imaging unit 11, an image processing unit 12, a communication unit 13, a vehicle speed sensor 14, and a steering angle sensor 15. A receiving device 2 includes a display unit 21, an image processing unit 22, and a communication unit 23. Note that the receiving device 2 does not necessarily have to include the display unit 21. The vehicle 1 and the receiving device 2 can communicate with each other via the communication units 13 and 23. The vehicle 1 may also be able to communicate with the receiving device 2 via the Internet. In other words, the vehicle 1 may be a connected car.

[0009] The imaging unit 11 is arranged so as to be able to capture images of the outside of the vehicle 1. The imaging unit 11 may capture images of traffic conditions around the vehicle 1. The traffic conditions may include information useful for the driving of the vehicle 1. The information useful for the driving of the vehicle 1 may be information indicating at least one of traffic lights, traffic signs, road markings, lanes, other vehicles, pedestrians, and obstacles. The vehicle speed sensor 14 detects the speed of the vehicle 1. The steering angle sensor 15 detects the steering angle of the vehicle 1 (for example, the steering angle of the wheels). Note that various existing aspects can be applied to the imaging unit 11, the vehicle speed sensor 14, and the steering angle sensor 15. Therefore, detailed descriptions of the imaging unit 11, the vehicle speed sensor 14, and the steering angle sensor 15 will be omitted.

[0010] The image processing unit 12 of the vehicle 1 performs predetermined image processing on the color image captured by the imaging unit 11. The image processing unit 12 transmits the image that has undergone image processing to the receiving device 2 via the communication unit 13.

[0011] The following technologies are known for transmitting video in real time. Monochrome processing is sometimes performed on video to maintain the video resolution and frame rate while reducing the wireless communication bandwidth. There is also a technology that uses a trained neural network to estimate color information related to monochrome video.

[0012] Combining these technologies makes it possible to achieve the following: First, a monochrome image is generated by performing monochrome processing on a color image captured by a first device. Next, the monochrome image is transmitted from the first device to a second device. Next, color information related to the monochrome image is estimated in the second device. As a result, the monochrome image transmitted from the first device can be viewed as a color image on the second device. In addition, the bandwidth used in wireless communication can be reduced.

[0013] However, the color information of the image estimated using a neural network is not necessarily the same as the original color. On the other hand, the image captured by the imaging unit 11 of the vehicle 1 includes objects whose colors are important. The objects whose colors are important may include at least one of traffic lights and traffic signs. Note that the objects whose colors are important may also include lines defining lane lines. It cannot be guaranteed that the color of the objects whose colors are important will be correctly estimated using a neural network.

[0014] In view of the above circumstances, the image processing unit 12 of the vehicle 1 performs the following image processing. In FIG. 2, an image IMG is an example of an image captured by the imaging unit 11. The image IMG includes a traffic light. The image processing unit 12 extracts an object whose color is important based on the image IMG. In this case, the image processing unit 12 may use a computational model that extracts an object whose color is important when the image IMG is input. The computational model may be a computational model using a neural network. The neural network may be a convolutional neural network (CNN).

[0015] The image processing unit 12 may use a plurality of computational models as a computational model for extracting objects whose colors are important. In this case, the image processing unit 12 may select one computational model from the plurality of computational models based on the imaging conditions of the imaging unit 11. The imaging conditions may include at least one of geography, weather, time of day, and a specific event. The geographical conditions may include at least one of road type and country. The specific event conditions may include a disaster. The plurality of computational models may be optimized so that objects whose colors are important can be extracted from images captured under the corresponding imaging conditions.

[0016] The image processing unit 12 may extract a traffic light from the image IMG as an object for which color is important. The image processing unit 12 determines a range R1 that includes the extracted traffic light. The size of the range R1 will be described later. The image processing unit 12 cuts out an image area corresponding to the range R1 from the image IMG as a color image cIMG. The image processing unit 12 may generate the color image cIMG by changing the pixel values of the image area of the image IMG outside the range R1 to values corresponding to black or white.

[0017] The video processor 12 performs monochrome processing on an image region of the image IMG that includes at least an area R2 other than the area R1. As a result, the video processor 12 generates a monochrome image mIMG. Note that Fig. 2 shows an example in which the monochrome image mIMG is generated by performing monochrome processing on the entire image IMG.

[0018] Here, the range R1 will be explained in more detail. The imaging period (in other words, the frame rate) of the imaging unit 11 is constant. When the vehicle 1 is traveling, the positional relationship between the vehicle 1 and an object whose color is important changes. Therefore, depending on the state of the vehicle 1, the position of an object included in an image at a first time may differ significantly from the position of the object included in an image at a second time after the first time. In other words, depending on the state of the vehicle 1, an object may move significantly within the image. If an object moves significantly within the image, it may take a long time to extract an object whose color is important. If it takes a long time to extract an object whose color is important, it becomes difficult to transmit video in real time.

[0019] The image processing unit 12 may determine the size of the range R1 based on the state quantities of the vehicle 1. The state quantities of the vehicle 1 may include at least one of the vehicle speed and steering angle of the vehicle 1. When the vehicle 1 moves forward, an object in front of the vehicle 1 approaches the vehicle 1. In this case, the size of the object in the image increases as the vehicle 1 approaches the object. The higher the vehicle speed of the vehicle 1, the faster the vehicle 1 approaches the object. When the vehicle 1 turns, the object in the image moves in the opposite direction to the turning direction of the vehicle 1. In this case, the amount of movement of the object in the image changes depending on the steering angle and the vehicle speed of the vehicle 1. When the vehicle speed of the vehicle 1 is constant, the larger the steering angle, the greater the amount of movement of the object in the image. Furthermore, when the steering angle is constant, the higher the vehicle speed of the vehicle 1, the greater the amount of movement of the object in the image. Therefore, the image processing unit 12 may increase the size of the range R1 as the vehicle speed of the vehicle 1 detected by the vehicle speed sensor 14 increases. Furthermore, the image processor 12 may increase the size of the range R1 as the steering angle detected by the steering angle sensor 15 increases.

[0020] The image processing unit 12 may perform the following processing. In FIG. 3 , the image processing unit 12 may extract an object whose color is important from an image IMG1 captured at time T1. The image processing unit 12 may set a range R1 for the image IMG1. The image processing unit 12 may generate a color image cIMG from the image IMG1 based on the range R1 of the image IMG1. The image processing unit 12 may not extract an object whose color is important from an image IMG2 captured at time T2. In this case, the image processing unit 12 may generate a color image cIMG from the image IMG2 based on the range R1 of the image IMG1. The image processing unit 12 may extract an object whose color is important from an image IMG3 captured at time T3. The image processing unit 12 may set a range R1 for the image IMG3. The image processing unit 12 may generate a color image cIMG from the image IMG3 based on the range R1 of the image IMG3. The image processing unit 12 may not extract an object whose color is important from image IMG4 captured at time T4. In this case, the image processing unit 12 may generate a color image cIMG from image IMG4 based on the range R1 of image IMG3. The image processing unit 12 may extract an object whose color is important from image IMG1 captured at time T5. The image processing unit 12 may set the range R1 for image IMG5. The image processing unit 12 may generate a color image cIMG from image IMG5 based on the range R1 of image IMG5.

[0021] According to the above-described process, the number of times that the process for extracting an object whose color is important is performed can be reduced. This makes it possible to secure time for appropriately performing the process for extracting an object whose color is important. The image processing unit 12 may determine the size of the range R1 based on the state quantity of the vehicle 1. Therefore, even if the number of times that the process for extracting an object whose color is important is reduced, the image processing unit 12 can generate a color image cIMG that includes an object whose color is important. As a result, it is possible to both extract an object whose color is important and transmit the image in real time.

[0022] The image processing unit 12 of the vehicle 1 will be further described with reference to the flowcharts of FIGS. 4 and 5. In FIG. 4, the image processing unit 12 acquires an image captured by the imaging unit 11 (step S101). The image processing unit 12 may select one computation model from a plurality of computation models based on the imaging conditions of the imaging unit 11 (step S102). The computation model is a computation model for extracting an object whose color is important from the image. The computation model may be a computation model using a neural network. The image processing unit 12 performs an extraction process to extract an object whose color is important from the image using the selected computation model (step S103). The operation shown in the flowchart of FIG. 4 may be repeated at a longer cycle than the operation shown in the flowchart of FIG. 5, which will be described later.

[0023] 5, the image processing unit 12 acquires an image captured by the imaging unit 11 (step S201). The image processing unit 12 determines whether or not the extraction process has been completed for the image acquired in the process of step S201 (step S202). If it is determined in the process of step S202 that the extraction process has been completed (step S202: Yes), the image processing unit 12 sets a range R1 for the image acquired in the process of step S201 based on the state quantities of the vehicle 1 (step S203). The image processing unit 12 stores the range R1 (step S204). Note that the image processing unit 12 cuts out an image region corresponding to the range R1 as a color image cIMG, and therefore the range R1 may also be referred to as a cut-out region.

[0024] If it is determined in the process of step S202 that the extraction process is not complete (step S202: No), the image processor 12 reads out the previous range R1 (step S205). Note that the previous range R1 refers to the range R1 stored in the process of step S204 in the previous cycle of the operation shown in the flowchart of FIG. 5.

[0025] After the processing of step S204 or S205, the image processor 12 cuts out an image region corresponding to range R1 from the image acquired in the processing of step S201 as a color image cIMG (step S206). In parallel with the processing of step S206, the image processor 12 performs monochrome processing on an image region of the image acquired in the processing of step S201 that includes at least range R2, which is a range other than range R1 (step S207). By the processing of step S207, a monochrome image mIMG is generated.

[0026] Thereafter, the video processing unit 12 encodes the color image cIMG and the monochrome image mIMG to transmit the video signal to the receiving device 2 (step S208). The video processing unit 12 transmits the encoded color image cIMG and monochrome image mIMG to the receiving device 2 via the communication unit 13 (step S209). The operation shown in the flowchart of Fig. 5 may be repeated at a cycle equal to or shorter than the imaging cycle of the imaging unit 11.

[0027] The receiving device 2 will be described with reference to the flowchart of FIG. 6. In FIG. 6, the receiving device 2 receives a color image cIMG and a monochrome image mIMG via the communication unit 23 (step S301). The video processing unit 22 of the receiving device 2 estimates color information related to the monochrome image mIMG using a trained neural network. The video processing unit 22 performs a coloring process on the monochrome image mIMG based on the estimated color information (step S302). The process of step S302 colorizes the monochrome image mIMG. The video processing unit 22 combines the colorized monochrome image mIMG with the color image cIMG to generate a single image (step S303). The image generated in the process of step S303 may be displayed on the display unit 21.

[0028] The receiving device 2 may be realized by at least one of a personal computer, a tablet terminal, and a smartphone. Alternatively, the receiving device 2 may be realized by a server device (e.g., a cloud server). In this case, the receiving device 2 does not need to include the display unit 21. The server device serving as the receiving device 2 may transmit the image generated by the processing of step S303 to the terminal device via the communication unit 23. The terminal device may be at least one of a personal computer, a tablet terminal, and a smartphone.

[0029] (Technical Effects) The image processor 12 of the vehicle 1 extracts an image region corresponding to range R1, which includes an object whose color is important, as a color image cIMG. Meanwhile, the image processor 12 generates a monochrome image mIMG by performing monochrome processing on an image region including at least range R2. The image processor 12 then transmits the color image cIMG and the monochrome image mIMG to the receiving device 2. The proportion of range R1 to the entire image is relatively small. Therefore, the data size of the color image cIMG is also relatively small. Therefore, the image processor 12 can maintain the resolution and frame rate of the image while reducing the bandwidth used for wireless communication. As a result, the image processor 12 can transmit the image to the receiving device 2 in real time.

[0030] The video processing unit 22 of the receiving device 2 colorizes the monochrome image mIMG and then combines it with the color image cIMG. At this time, the colors of the objects whose colors are important in the color image cIMG are the same as the colors in the original image. Therefore, it is possible to ensure that the colors of the objects whose colors are important in the image combined by the video processing unit 22 are the same as the colors in the original image. Therefore, according to this embodiment, color video can be appropriately recorded and transmitted.

[0031] Aspects of the invention derived from the above-described embodiments will be described below.

[0032] An image processing method according to one aspect of the invention is an image processing method for processing color image captured by an imaging means for capturing images of traffic conditions around a vehicle, the image processing method including: a retaining step of retaining, in color, a first image portion of the image having a first range smaller than the range captured by the imaging means; and a converting step of converting, to monochrome, a second image portion of the image including at least a second range other than the first range. In the above embodiment, the "range R1" corresponds to an example of the "first range," the "range R2" corresponds to an example of the "second range," the "color image cIMG" corresponds to an example of the "first image portion," and the "monochrome image mIMG" corresponds to an example of the "second image portion."

[0033] The image processing method may include a determining step of determining the size of the first range based on a state quantity of the vehicle. Here, the state quantity may be a speed of the vehicle. In the determining step, the size of the first range may be larger as the vehicle speed increases. Furthermore, the state quantity may be a steering angle of the vehicle. In the determining step, the size of the first range may be larger as the steering angle increases.

[0034] The image processing method may include a transmission step of transmitting the first image portion maintained in color and the second image portion converted to monochrome to an external device different from the vehicle, and a synthesis step of the external device colorizing the second image portion converted to monochrome and synthesizing the colorized second image portion with the first image portion maintained in color.

[0035] The present invention is not limited to the above-described embodiments. The present invention can be modified as appropriate within the scope of the claims and the spirit or concept of the invention as can be read from the entire specification. Image processing methods involving such modifications are also included within the technical scope of the present invention. [Explanation of symbols]

[0036] 1...vehicle, 2...receiving device, 11...imaging unit, 12, 22...image processing unit, 13, 23...communication unit, 14...vehicle speed sensor, 15...steering angle sensor, 21...display unit

Claims

1. 1. An image processing method for processing color images captured by an imaging means for capturing images of traffic conditions around a vehicle, comprising: a holding step of holding, in color, a first image portion of the image, the first image portion having a first range smaller than the range captured by the imaging means; a conversion step of converting a second image portion of the image, the second image portion including at least a second range other than the first range, into monochrome; a transmitting step of transmitting the first image portion maintained in color and the second image portion converted to monochrome to an external device other than the vehicle; a combining step in which the external device colorizes the second image portion converted to monochrome and combines the colorized second image portion with the first image portion maintained in color; A video processing method comprising:

2. The image processing method according to claim 1 , further comprising a step of determining the size of the first range based on a state quantity of the vehicle.

3. the state quantity is a vehicle speed of the vehicle, In the determining step, the greater the vehicle speed, the greater the size of the first range.

3. The video processing method according to claim 2.

4. the state quantity is a steering angle of the vehicle, In the determination step, the larger the steering angle, the larger the size of the first range.

4. The video processing method according to claim 2 or 3.

Images