Program, device and method for generating useful image stream from original image stream

Abstract

The useful image stream generation program of the present invention enables a computer to function as the following units: a coding parameter extraction unit that extracts coding parameters of each macroblock for each frame from the original image stream; a macroblock screening unit that screens macroblocks having coding parameters that satisfy pre-set conditions as useful macroblocks; and a useful image stream generation unit that generates a useful image stream by combining multiple frames of the original image stream that are temporally synchronized with the frames of the coding parameters in the useful macroblocks in a time series.

Description

Technical Field

[0001] The present invention relates to a technology for generating a useful video stream with reduced capacity from a video stream captured by a camera.

[0002] This application is based on and claims the benefit of priority from Japanese Patent Application No. 2019-91895 filed in Japan on May 15, 2019, the contents of which are incorporated herein by reference. Background Art

[0003] As a video stream encoding technology, the representative MPEG (Moving Picture Experts Group) is known. According to this technology, in order to improve the compression efficiency of the video stream, a predicted image of the current frame is generated based on the previous and / or next frames, and the difference (error) image between the input image and the predicted image is encoded using an "inter-frame prediction" method.

[0004] The video stream may be a video stream captured by a user's home camera or a video stream captured by a fixed surveillance camera. In recent years, a device for generating video streams has been a vehicle-mounted recorder equipped with a camera that captures the view from the vehicle. The vehicle-mounted recorder continuously stores the video stream captured while the vehicle is traveling.

[0005] Figure 1 This is a system configuration diagram including a terminal device as an in-vehicle recorder installed in a vehicle.

[0006] like Figure 1 As shown, the terminal 1 as a vehicle-mounted recorder uses a camera to shoot the image of the vehicle's traveling direction and stores the image stream in a storage unit (memory, disk). The terminal 1 sends the image stream to the data center 2 via a wireless network.

[0007] In addition, the terminal 1 can obtain vehicle travel information via CAN (Controller Area Network), and the vehicle travel information can also be sent to the data center 2 together with the video stream.

[0008] The data center 2 can analyze the image stream and vehicle driving information received from the terminal 1. For example, it can detect and / or identify objects reflected in the image stream and establish a correspondence with the vehicle driving information.

[0009] In the past, for example, Patent Document 1 discloses a technique for encoding image data divided into a plurality of regions at a different encoding rate for each region using a plurality of cameras installed in a vehicle. According to this technique, a resolution required according to the importance is calculated for each region, and the image data is encoded at a encoding rate corresponding to the importance.

[0010] For example, Patent Document 2 discloses a technology in which, when the value of the acceleration of the vehicle in a specific direction exceeds a predetermined value, the vehicle recorder outputs image data acquired by a plurality of cameras installed in the vehicle.

[0011] Furthermore, for example, Patent Document 3 discloses a technology for a remote operation support system that reduces the amount of data for an image captured by a camera worn by an operator. According to this technology, it is evaluated whether there is a region in the image frame where an object moves a lot, and if there is no region where the object moves a lot, it is considered unimportant and deleted.

[0012] Prior art literature

[0013] Patent Literature

[0014] Patent Document 1: Japanese Patent Application Publication No. 2010-263500

[0015] Patent Document 2: Japanese Patent Application Publication No. 2013-218433

[0016] Patent Document 3: Japanese Patent Application Publication No. 2018-082333 Summary of the invention

[0017] Problems to be solved by the invention

[0018] According to the technology described in Patent Document 1, it is necessary to identify regions for image data before encoding. When the camera outputs encoded image data, decompression is required, which makes the processing complicated. In addition, although the encoding rate is different for each region, since all camera images are encoded, the overall data reduction effect is small.

[0019] According to the technology described in Patent Document 2, since only the acceleration of the vehicle is used as a determination factor for outputting image data, there is a disadvantage that image data is not output when the vehicle is traveling at a constant acceleration.

[0020] According to the technology described in Patent Document 3, the size of the movement of an object in an image is evaluated and encoded. Therefore, there is a disadvantage that it is not possible to evaluate the encoding parameters of the macroblocks for the encoded video data and determine the video data to be reduced.

[0021] In contrast, the inventors of the present application consider that if a useful image stream containing only useful parts required for image analysis of the original image stream can be generated, the data capacity to be stored or transmitted can be reduced. In addition, it is considered that the device receiving the useful image stream can also reduce the processing burden for detecting and / or identifying objects reflected in the useful image stream.

[0022] Therefore, an object of the present invention is to provide a program, an apparatus, and a method for generating a useful video stream containing only useful parts required for video analysis of an original video stream.

[0023] Technical means of solving problems

[0024] The program according to the present invention causes a computer to function as a device for generating a useful video stream from a captured and encoded original video stream, wherein:

[0025] The program causes the computer to function as the following means:

[0026] A coding parameter extraction unit, extracting coding parameters of each macroblock for each frame from the original image stream;

[0027] a macroblock screening unit, screening the macroblocks having encoding parameters satisfying a preset condition as useful macroblocks; and

[0028] The useful image stream generating unit generates a useful image stream by combining the encoding parameter frame in the useful macroblock with a plurality of temporally synchronized frames of the original image stream in time series.

[0029] The program of the present invention preferably enables the useful image stream generating unit to perform the following function: in addition to the image stream obtained by combining the frames of the original image stream synchronized with the frames of the encoding parameter in time series, the image stream obtained by combining the frames of the encoding parameter in time series is generated as the useful image stream.

[0030] The program of the present invention preferably causes the useful image stream generating unit to function as follows: including a frame number and / or a time code for each frame as a useful image stream.

[0031] The program of the present invention preferably causes the useful image stream generating unit to function to include useful macroblocks and distribution information of the useful macroblocks for each frame as a useful image stream.

[0032] The program of the present invention preferably causes the useful image stream generating unit to function to include frames having useful macroblocks as useful image streams in units of GOP (Group Of Pictures) including frames.

[0033] The program of the present invention preferably causes the useful video stream generating unit to function to delete or compress a GOP of a frame not corresponding to the encoding parameter in the useful macroblock at a high compression rate of a predetermined ratio or more.

[0034] In the program of the present invention, it is preferred that the original video stream is output from a camera mounted on a mobile object.

[0035] The useful image stream includes images of only objects that appear or disappear among the objects photographed by the camera.

[0036] In the program of the present invention, preferably, the encoding is performed based on MPEG (Moving Picture Experts Group),

[0037] The encoding parameters include the size and direction of the motion vector of forward prediction (inter-frame), the coding amount or the size of the quantization step of intra-frame prediction (intra-frame).

[0038] In order to select useful macroblocks whose time variation is larger than a predetermined value, the program of the present invention preferably enables the macroblock selection unit to perform the following function: performing the following processing as the preset condition for each macroblock:

[0039] The motion vector of the forward prediction has a size greater than a predetermined length and a direction greater than a predetermined angle relative to the convergence point on the horizon;

[0040] The intra-picture prediction code amount is greater than or equal to the first code amount; or

[0041] The quantization step size is larger than a predetermined size.

[0042] The program of the present invention preferably causes the useful video stream generating unit to function to determine that a frame having an intra-screen prediction code amount equal to or greater than a second code amount is a frame in which the moving object is stationary, and exclude the frame from the useful video stream.

[0043] The program of the present invention preferably causes the macroblock selection unit to perform the following functions: generate a coding parameter map in which the useful macroblocks are combined with spatial directions according to distribution information,

[0044] In the encoding parameter map, the higher the degree to which the encoding parameter satisfies the preset condition, the darker the tone of the macroblock portion is displayed.

[0045] Preferably, in the process of the present invention,

[0046] The original image stream is encoded,

[0047] The program causes the encoding parameter extraction unit to function to decompress the original video stream in a simple manner and extract the encoding parameter of each macroblock for each frame.

[0048] A useful stream generating device generates a useful image stream from a captured and encoded original image stream, wherein the device comprises: a coding parameter extraction unit for extracting coding parameters of each macroblock for each frame from the original image stream; a macroblock screening unit for screening the macroblocks having coding parameters that satisfy preset conditions as useful macroblocks; and a useful image stream generating unit for generating a useful image stream by combining a plurality of frames of the original image stream that are temporally synchronized with the frames of the coding parameters in the useful macroblocks in a time series.

[0049] A method for generating a useful image stream is provided, which generates a useful image stream from a captured and encoded original image stream, wherein the method comprises the following steps: a first step of extracting encoding parameters of each macroblock for each frame from the original image stream; a second step of screening the macroblocks having encoding parameters that satisfy preset conditions as useful macroblocks; and a third step of generating a useful image stream by combining a plurality of frames of the original image stream that are time-synchronized with the frames of the encoding parameters in the useful macroblocks in a time series.

[0050] Effects of the Invention

[0051] According to the program, device, and method of the present invention, it is possible to generate a useful video stream containing only useful parts required for video analysis of an original video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a diagram showing a system including a terminal serving as an in-vehicle recorder installed in a vehicle.

[0053] Figure 2 It is a block diagram showing a terminal according to one embodiment of the present invention.

[0054] Figure 3 This is an explanatory diagram showing a coding parameter extraction unit in a terminal according to an embodiment of the present invention.

[0055] Figure 4A This is a diagram showing a motion vector for each macroblock when an object projected into an original video stream changes at a constant speed in one embodiment of the present invention.

[0056] Figure 4B This is a diagram showing a motion vector for each macroblock when an object projected into an original video stream changes suddenly in one embodiment of the present invention.

[0057] Figure 5 It is a diagram showing a coding parameter diagram according to one embodiment of the present invention.

[0058] Figure 6This is a diagram showing an example in which only the distribution portion of macroblocks satisfying a preset condition is generated as a coding parameter map in one embodiment of the present invention.

[0059] Figure 7 This is a diagram showing the operation of a useful video stream generating unit in a terminal according to one embodiment of the present invention. DETAILED DESCRIPTION

[0060] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0061] The present invention can generate a "useful image stream" containing only useful parts required for image analysis from an "original image stream".

[0062] The “original video stream” is, for example, a coded video stream captured by a camera mounted on a moving object such as a vehicle.

[0063] The “useful image stream” includes, for example, only images required for object detection or object recognition, or for analyzing the driving state (speed or acceleration) of a moving object.

[0064] Generally speaking, a useful image stream is an image stream that only includes surrounding objects that suddenly appear or disappear from the surrounding objects that can be seen by the camera. Of course, it is not limited to this.

[0065] Therefore, the useful video stream is a video stream obtained by reducing the capacity of the video stream to be transmitted or stored from the perspective of the original video stream.

[0066] Figure 2 It is a functional structure diagram of the terminal in the present invention.

[0067] According to the embodiment of the present invention, the terminal 1 is a vehicle-mounted recorder equipped with a camera C, which is installed inside the vehicle. The shooting direction D of the camera C is not limited to the front of the vehicle's traveling direction, but may also be the rear or side.

[0068] In addition, the terminal 1 can communicate with the data center 2 via any wireless network. The data center 2 has a function of analyzing images based on useful image streams. Of course, it is not limited to a data center, and an edge computing device may also be used.

[0069] Reference Figure 2 The terminal 1 has a camera C, an original image stream storage unit 10, a coding parameter extraction unit 11, a macroblock screening unit 12, a useful image stream generation unit 13, an image stream transmission unit 14, and a communication interface 15. These functional components except the camera C are implemented by executing a program stored in a memory to enable a computer (processor) mounted on the device to function. In addition, the processing flow of these functional components can be regarded as a useful image stream generation method of the device.

[0070] Reference Figure 2 , the camera C captures the image in the shooting direction D and outputs the original image stream. The original image stream may be an image stream encoded inside the camera C. In the case of outputting image data not encoded by the camera C, the unencoded image data may be encoded by a device or software other than the camera C.

[0071] The encoding method may be a standard format such as H.264 or H.265 based on MPEG, or a non-standard format. In addition, the spatial resolution, temporal resolution (frame rate), and encoding rate (bit rate) may be arbitrary.

[0072] [Original video stream storage unit 10]

[0073] The original video stream storage unit 10 temporarily stores the original video stream captured by the camera C. The original video stream is, for example, an MPEG-encoded video stream.

[0074] The original video stream storage unit 10 outputs the original video stream to the encoding parameter extraction unit 11 at an arbitrary timing.

[0075] [Coding parameter extraction unit 11]

[0076] The encoding parameter extraction unit 11 simply decompresses (decodes) the encoded original image stream and interprets (parses) the bit stream. Here, simply decompressing only extracts the encoding parameters from the encoded original image stream. That is, the original image stream is not completely decompressed, nor is it restored to a visible image frame. For example, in the case where the encoding parameters show differences, it is only necessary to restore them to the original values ​​so that the encoding parameters can be processed.

[0077] Figure 3 This is a diagram for explaining the operation of the coding parameter extraction unit 11 in the terminal 1 according to the present embodiment.

[0078] like Figure 3 As shown, the encoded original video stream is composed of a sequence header and GOP (Group Of Pictures) data.

[0079] The sequence header describes the horizontal and vertical numbers of pixels in the frame, etc.

[0080] GOP data is composed of a GOP header and a plurality of picture data (a set of I, P, and B picture data) in a series. The GOP header includes a time code indicating the time when the picture in the group is first displayed, etc. The picture data represents one frame (picture).

[0081] The image data is composed of an image header and slice data (slice 1 to slice 68).

[0082] like Figure 3 As shown, one image data is composed of 1088 pixels in the vertical direction (68 slices) x 1440 pixels in the horizontal direction (90 macroblocks). Slices are obtained by cutting one image into strips.

[0083] Each slice is composed of a slice header and 90 macroblock data (MB1 to MB90).

[0084] Each macroblock is a square pixel block of 16 pixels×16 rows.

[0085] Each macroblock contains a macroblock address, macroblock type, quantization step, motion vector, and block data. Of course, the structure of the original image stream is not limited to this, and can also be set arbitrarily according to the horizontal and vertical pixel numbers, slice numbers, and macroblock sizes of the image data.

[0086] The coding parameter extraction unit 11 extracts coding parameters of each macroblock for each frame from the original video stream.

[0087] The encoding parameter is any of the following.

[0088] (1) The size and direction of the motion vector for forward prediction (Inter frame)

[0089] (2) Intra-frame prediction block type coding amount

[0090] (3) Quantization step size

[0091] [Macroblock selection unit 12]

[0092] The macroblock selection unit 12 selects macroblocks having coding parameters that satisfy a preset condition as useful macroblocks. Here, the "preset condition" is a case where the time variation is greater than a prescribed value. Each macroblock is selected according to any one of the following preset conditions, for example, <1>, <2>, and <3>.

[0093] <1> The motion vector of the forward prediction is larger than a predetermined length and has a predetermined angle in the direction of the convergence point on the horizon.

[0094] <2> The intra-frame prediction code amount is equal to or greater than the first code amount

[0095] <3> The quantization step size is larger than a predetermined size.

[0096] The macroblock filter 12 extracts a relatively rapid change area. "Rapid change" includes a situation where surrounding objects visible from the camera suddenly change, and a situation where the running state of a moving object (eg, a vehicle) equipped with the camera suddenly changes.

[0097] <1. Selection of macroblocks based on forward predicted motion vectors>

[0098] Figure 4A and Figure 4B This is an explanatory diagram showing the motion vector for each macroblock.

[0099] Figure 4A It indicates that the object projected into the original image stream changes at a constant speed.

[0100] Figure 4B This indicates a situation where an object reflected in the original image stream changes suddenly.

[0101] Respectively Figure 4A and Figure 4B The image frames of the original image stream and the motion vectors of each macroblock are shown in FIG.

[0102] Reference Figure 4A , in the dotted elliptical area 4D, the object's motion vector is oriented in the same direction as the vehicle's travel direction. Figure 4A The following distribution of motion vectors is observed.

[0103] The direction of the motion vector is toward the convergence point on the horizon and changes less with time.

[0104] The size of the motion vector is smaller the closer it is to the convergence point on the horizon, and larger the farther it is from the convergence point on the horizon.

[0105] For example, when the front vehicle is also traveling at a constant speed relative to the vehicle, the encoding parameters of the area 4D where the front vehicle is reflected do not satisfy the above-mentioned "pre-set conditions". Figure 4A No useful macroblocks can be filtered out.

[0106] Reference Figure 4B In region 4L, the motion vector of the object is oriented in the same direction as the vehicle's traveling direction. On the other hand, in region 4R, the motion vector of the object is oriented in a direction different from the vehicle's traveling direction. Figure 4B In region 4R, the following distribution of motion vectors is observed.

[0107] The direction of the motion vector is different from the convergence point on the horizon and changes greatly with time.

[0108] The magnitude of the motion vector is large regardless of the distance from the convergence point on the horizon.

[0109] For example, as shown in area 4R, when the front vehicle suddenly changes lanes or slows down relative to the vehicle, the coding parameters of the area where the front vehicle is reflected meet the coding parameters of the above-mentioned "pre-set conditions". Figure 4B Area 4R screens out useful macroblocks.

[0110] By utilizing this characteristic, it is possible to estimate areas with rapid changes in the original video stream as useful macroblocks.

[0111] As another embodiment, it is preferable to adaptively change the selection criteria (preset conditions) of the size of the motion vector according to the following situations (a), (b), and (c), for example.

[0112] (a) When calculating a motion vector by referring to a frame two or more frames ago

[0113] (b) When the motion vector is calculated by referring to the previous frame

[0114] (c) When referring to multiple frames

[0115] Here, in the case of (a), the size of the motion vector when referring to the frame two frames before is compared with the size of the motion vector in the case of (b), and is approximately twice as large.

[0116] In addition, the size of the motion vector after encoding at 30 frames / second is about 1 / 3 of the size of the motion vector after encoding at 10 frames / second.

[0117] Therefore, the determination of the preset conditions can also be adaptively changed according to the degree of comparison.

[0118] <2. Selection of macroblocks based on the amount of coding by intra-frame prediction>

[0119] The amount of code for a macroblock predicted within a screen increases when there is an edge (outline of an object) within the screen, and decreases when the screen is flat.

[0120] That is, in Figure 4A In the case of , since the road area is flat, more macroblocks predicted by intra-frame with less coding amount are observed. Figure 4B In the case of , many intra-screen predicted macroblocks with a large amount of coding are observed at the edge of the vehicle that wants to turn right in area 4R.

[0121] By utilizing this characteristic, it is possible to estimate the area where the object exists in the original image stream as a useful macroblock.

[0122] <3. Selection of macroblocks based on the number of intra-frame prediction macroblocks>

[0123] The number of macroblocks predicted within the picture increases when a sharp change occurs within the picture, and decreases when the change is small.

[0124] That is, in the following Figure 6 In the case of the original video stream on the left, since there are fewer changes in the picture, fewer macroblocks are observed to be predicted in the picture. Figure 6 In the case of the original video stream on the right side, since the image of the vehicle reappears, more macroblocks predicted within the screen are observed.

[0125] By utilizing this characteristic, it is possible to estimate areas with rapid changes in the original video stream as useful macroblocks.

[0126] <4. Macroblock selection based on quantization step size>

[0127] According to the adaptive quantization method, the quantization step size of the macroblock is adaptively changed to improve compression efficiency. For the original image stream, the quantization step size is reduced because human vision is more sensitive in areas with less changes. On the other hand, the quantization step size is increased because human vision is less sensitive in areas with more intense changes.

[0128] That is, in Figure 4A In the case of , an increase in macroblocks with smaller quantization step sizes is observed. On the other hand, in Figure 4B An increase in macroblocks with a larger quantization step size is observed in the wheel portion of the front vehicle on the right side.

[0129] By utilizing this characteristic, it is possible to estimate areas with rapid changes in the original video stream as useful macroblocks.

[0130] In addition, as another embodiment, the macroblock screening unit 12 may generate a "coding parameter map" by combining useful macroblocks with spatial directions based on distribution information.

[0131] Figure 5 A diagram showing a coding parameter map in an embodiment of the present invention.

[0132] In the "coding parameter map", the higher the degree to which the coding parameter satisfies the preset condition, the darker the tone of the part of the macroblock can be displayed.

[0133] In this way, in the encoding parameter map, for example, the distribution portion of the macroblock can be displayed in a color according to the size and direction of the motion vector of the macroblock, the encoding amount, or the hue of the quantization step.

[0134] (1) For example, the larger the size of the motion vector of the forward prediction is, and the larger the displacement is from the direction toward the convergence point on the horizon, the darker the color will be displayed in the distribution portion of the macroblock.

[0135] (2) For example, when the amount of intra-prediction coding is larger (the macroblock is in the intra-prediction coding mode), a darker color is displayed in the distribution portion of the macroblock.

[0136] (3) For example, the smaller the quantization step size of a macroblock is, the darker the color will be displayed in the distribution portion of the macroblock.

[0137] like Figure 5 As shown in the coding parameter diagram of , in the region 5L where the change is constant and there are few sudden changes, the amount of code for intra-screen prediction is small. On the other hand, in the region 5R where there is a sudden change, the amount of code for intra-screen prediction is large.

[0138] Furthermore, as another embodiment, it is preferable to exclude the region without temporal change in the frame from the extraction target of the coding parameter.

[0139] Figure 6 This is a diagram for explaining the case where only the distribution portion of macroblocks satisfying a preset condition is generated as a coding parameter map.

[0140] like Figure 6 As shown in the lower right figure of , only the area with a sharp change is extracted from the original image stream. Since the encoding parameter map does not include macroblocks that do not meet the pre-set conditions, the overall data volume can be greatly reduced.

[0141] For example, due to the installation position of the onboard recorder in the vehicle, there is sometimes a situation where an area such as the dashboard is reflected in the original image stream captured by the camera. If such an area is fixed in advance, it can be blocked by the factory setting. Of course, the user can also use the GUI (Graphic User Interface) to set the area he wants to block.

[0142] In the case where a region in a frame that does not change over time is flat, for example, "skip mode" is selected as the coding mode for the macroblock. Such a region can be set as a non-extraction target for coding parameters. For example, the region of the instrument panel in the original image stream is flat, and by selecting "skip mode" as the coding mode for the macroblock, the amount of coding for intra-frame prediction is reduced.

[0143] In this way, for each macroblock, the change status within a specified time width is observed. Compared with other areas in the picture, the coding amount of the block predicted in the picture is obviously reduced. When the frequency of occurrence of the block in the skip mode is high, it is effective to delete this area.

[0144] [Useful video stream generation unit 13]

[0145] The useful video stream generation unit 13 generates a useful video stream by combining a plurality of frames of the original video stream that are synchronized in time with the frames of the coding parameters in the useful macroblocks in a time series (see Figure 7 ).

[0146] Furthermore, the useful video stream generating unit 13 may combine the frames of the encoding parameter in time series in addition to combining the frames of the original video stream synchronized with the frames of the encoding parameter in time series.

[0147] like Figure 4A , Figure 4B or Figure 5 As shown in FIG. 1 , the “coding parameter frame” refers to a frame in which the coding parameters are distributed according to the distribution of macroblocks. That is, the coding parameters extracted by the coding parameter extraction unit 11 are mapped.

[0148] "Frames of the original video stream synchronized with the frames of the encoding parameters" literally means using each frame of the original video stream.

[0149] By using the code amount of each frame, it is possible to filter only the frames with rapid changes in the original video stream.

[0150] For example, when the original video stream changes at a constant speed and does not change sharply, there is motion in the picture, and the number of macroblocks predicted in the forward direction increases. Therefore, there is a tendency that the number of macroblocks predicted in the picture decreases, and the amount of code for the entire frame decreases. For example, when the original video stream of surrounding vehicles traveling at a constant speed is projected from the camera of the own vehicle, the amount of code for the entire frame decreases.

[0151] For example, in H.264, which performs encoding at a fixed bit rate of about 6 Mbit per second, the encoding amount of an I (Intra) frame in a running original video stream varies from 500K to 800K bits.

[0152] (2) On the other hand, when there is a sudden change in the original image stream, the number of macroblocks predicted within the screen increases. Therefore, there is a tendency that the number of macroblocks predicted in the forward direction decreases, and the amount of code for the entire frame increases. For example, when the original image stream of surrounding vehicles that has changed suddenly is projected from the camera of the own vehicle, the amount of code for the entire frame increases.

[0153] (3) Furthermore, for the original video stream, when there is no motion in the picture (e.g., when the picture is stopped), there is a tendency that the number of macroblocks predicted in the forward direction decreases, the number of macroblocks predicted in the intra-picture direction increases, and the overall coding amount of the frame increases. For example, for the original video stream that is stopped and unchanged, the coding amount of an I (intra-frame) frame becomes more than 1M bits.

[0154] In this case, by setting the threshold value to 1 Mbit as the code amount of the intra-image coded frame, it is possible to determine the running / stopping state.

[0155] In this way, by observing the code amount of the frame, it is possible to estimate a sudden change in the driving condition, etc. (see Figure 7 ).

[0156] As another embodiment, instead of observing the coding amount of each frame, the number of macroblocks predicted in each frame can be counted to estimate the sudden change in driving conditions. Frames with a predetermined number of macroblocks predicted in the frame can be judged as having a sudden change in the object reflected in the frame.

[0157] For example, when a vehicle enters / exits a tunnel, or when visibility suddenly deteriorates due to heavy rain or smoke, there is a tendency for most of the frame area of ​​the original image stream captured by the camera of the vehicle recorder to become macroblocks predicted within the screen.

[0158] Figure 7 The second diagram shows the operation of the useful video stream generating unit 13 in the terminal according to the embodiment of the present invention.

[0159] <Selection of frames based on the amount of code>

[0160] The useful video stream generating unit 13 determines that a frame whose code amount of an I (intra) frame is equal to or larger than a second code amount (eg, 1 Mbit) means that the moving object being photographed has stopped, and excludes the frame from the useful video stream.

[0161] Therefore, the number of frames of the useful image stream can be reduced.

[0162] At this time, it is preferable to set the second code amount, which is the threshold value for the above determination, to different values ​​depending on the case of CBR (fixed bit rate) coding and the case of VBR (variable bit rate) coding. It is necessary to change the basis for evaluating the absolute code amount or the difference in the relative code amount with other predictive coding frames. In this way, it is preferable to adaptively control the second code amount by considering the difference in the structure of the original image stream, etc.

[0163] As another embodiment, regarding the frames to be excluded from the time series frames of the original video stream, the compression rate (coding rate) may be increased without excluding the frames themselves, that is, as long as the frames having at least useful macroblocks can be sufficiently played.

[0164] The useful video stream generation unit 13 generates a useful video stream including a frame number and / or a time code for each frame.

[0165] In addition, it is preferable that the useful video stream generation unit 13 includes useful macroblocks and distribution information of these macroblocks for each frame as a useful video stream. Therefore, it is possible to identify the useful area of ​​each frame in the useful video stream. In other words, it is possible to reduce macroblocks in non-useful areas.

[0166] The data center 2 that receives the useful video stream can play the useful video stream by receiving the frame number, time code, and macroblock distribution information together with the useful video stream.

[0167] As another embodiment, it is preferable that the useful video stream generating unit 13 includes frames having useful macroblocks in units of GOP (Group Of Pictures) as the useful video stream, thereby being able to gather video frame groups to be transmitted.

[0168] In addition, the GOP of the frame that does not correspond to the encoding parameters in the useful macroblock can be deleted or compressed at a high compression rate of more than a predetermined ratio.

[0169] In addition, as another embodiment, the useful image stream generation unit 13 may include a user interface for instructing the start and end of observation. Frames during a period when the vehicle is parked and the entire screen does not change can be excluded from the useful image stream.

[0170] [Video stream sending unit 14]

[0171] The video stream transmitting unit 14 transmits the useful video stream output from the useful video stream generating unit 13 to the data center 2 via the communication interface 15 .

[0172] The useful video stream is transmitted at an arbitrary timing. For example, it may be transmitted at the time when it is output from the useful video stream generating unit 13, or it may be temporarily buffered and transmitted at the time when a predetermined capacity is reached.

[0173] Furthermore, when the useful video stream is composed of frames of encoding parameters, an original video stream composed of original video frames corresponding to the frames of the encoding parameters may be transmitted synchronously or separately.

[0174] Furthermore, the frame number and / or time code, useful macroblocks, and distribution information of these macroblocks can be sent synchronously or asynchronously with the useful image stream, so that the data center can play useful macroblocks from the useful image stream.

[0175] As another embodiment, the video stream transmitting unit 14 preferably transmits the driving state information to which the time code is added together with the useful video stream. The encoding parameter and the driving state information can be associated based on the time code. In this case, the association is preferably performed in a predetermined time span of the time code.

[0176] As described above, according to the program, device, and method of the present invention, it is possible to generate a useful video stream including only useful parts required for video analysis of an original video stream.

[0177] Therefore, the encoding parameters are used to select only useful image frames and image areas from the original image stream to be encoded, thereby reducing the overall data volume. In particular, when processing object detection or object recognition in a data center, it is also possible to limit the image stream that should be the target.

[0178] For the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical ideas and viewpoints of the present invention. The above description is only an example and is not intended to be limiting. The present invention is only limited by the claims and their equivalents.

[0179] Description of Reference Numerals

[0180] 1 Terminal

[0181] 10 Original video stream storage unit

[0182] 11 Coding parameter extraction unit

[0183] 12 Macroblock filtering unit

[0184] 13 Useful image stream generation unit

[0185] 14 Video stream sending unit

[0186] 15 Communication Interface

[0187] 2 Data Center

Claims

1. A storage medium storing a program for causing a computer to function as a device for generating a useful video stream from a captured and encoded original video stream, wherein: The program causes the computer to function as the following means: A coding parameter extraction unit, extracting coding parameters of each macroblock for each frame from the original image stream; A macroblock screening unit, screening the macroblocks having encoding parameters satisfying a preset condition as useful macroblocks; as well as a useful image stream generating unit for generating a useful image stream by combining a plurality of frames of the original image stream that are synchronized in time with the frames of the encoding parameters in the useful macroblock in a time series; In the storage medium, as the encoding parameter extracted from the macroblock satisfying the preset condition, at least the angle of the motion vector toward the convergence point on the horizon is used, The motion vector is a motion vector of inter-frame prediction in the macroblock.

2. The storage medium according to claim 1, wherein: The program causes the useful video stream generating unit to function as follows: in addition to the video stream obtained by time-sequentially combining frames of the original video stream synchronized with the frames of the encoding parameter, the video stream obtained by time-sequentially combining the frames of the encoding parameter is generated as the useful video stream.

3. The storage medium according to claim 1 or 2, wherein: The program causes the useful image stream generating unit to function as follows: including a frame number and / or a time code for each frame as the useful image stream.

4. The storage medium according to claim 3, wherein: The program causes the useful image stream generating unit to function to include useful macroblocks and distribution information of the useful macroblocks for each frame as the useful image stream.

5. The storage medium according to claim 3, wherein: The program causes the useful image stream generating unit to function to include frames having useful macroblocks in units of GOP (Group Of Pictures) as the useful image stream.

6. The storage medium according to claim 5, wherein: The program causes the useful video stream generating unit to function to delete or compress a GOP of a frame that does not correspond to the encoding parameter in the useful macroblock at a compression rate higher than a predetermined ratio.

7. The storage medium according to claim 1, wherein: The original video stream is output from a camera mounted on a mobile object. The useful image stream includes images of only objects that appear or disappear among the objects photographed by the camera.

8. The storage medium according to claim 7, wherein: The encoding is based on MPEG (Moving Picture Experts Group). The encoding parameters include the size and direction of the motion vector of forward prediction (inter-frame), the coding amount or the size of the quantization step of intra-frame prediction (intra-frame).

9. The storage medium according to claim 8, wherein: In order to select useful macroblocks whose time variation is larger than a predetermined value, the program causes the macroblock selection unit to function as follows, and uses the following contents as the preset conditions for each macroblock: The motion vector of the forward prediction has a size greater than a predetermined length and a direction greater than a predetermined angle relative to the convergence point on the horizon; The intra-picture prediction code amount is greater than or equal to the first code amount; or The quantization step size is larger than a predetermined size.

10. The storage medium according to claim 9, wherein: The program causes the useful video stream generating unit to function to determine that a frame in which the intra-frame prediction code amount is equal to or greater than a second code amount is a frame in which the moving object is stationary, and exclude the frame from the useful video stream.

11. The storage medium according to claim 1, wherein: The program causes the macroblock selection unit to perform the following functions: generate a coding parameter map in which the useful macroblocks are joined in a spatial direction according to distribution information; In the encoding parameter map, the higher the degree to which the encoding parameter satisfies the preset condition, the darker the tone of the macroblock portion is displayed.

12. The storage medium according to claim 1, wherein: The original image stream is encoded, The program causes the encoding parameter extraction unit to function to simply decompress the original video stream and extract the encoding parameter of each macroblock for each frame.

13. A useful image stream generating device, which generates a useful image stream from a captured and encoded original image stream, wherein: have: A coding parameter extraction unit, extracting coding parameters of each macroblock for each frame from the original image stream; A macroblock screening unit, screening the macroblocks having encoding parameters satisfying a preset condition as useful macroblocks; as well as a useful image stream generating unit for generating a useful image stream by combining a plurality of frames of the original image stream that are synchronized in time with the frames of the encoding parameters in the useful macroblock in a time series; In the useful image stream generating device, at least the angle of the motion vector toward the convergence point on the horizon is used as the encoding parameter extracted from the macroblock that satisfies the preset condition. The motion vector is a motion vector of inter-frame prediction in the macroblock.

14. A method for generating a useful image stream from a captured and encoded original image stream, wherein: The steps include: The first step is to extract the encoding parameters of each macroblock for each frame from the original image stream; The second step is to select the macroblocks having encoding parameters satisfying the preset conditions as useful macroblocks; as well as The third step is to generate a useful image stream by combining a plurality of frames of the original image stream that are synchronized in time with the frames of the encoding parameters in the useful macroblock in a time series. In the useful image stream generation method, at least the angle of the motion vector toward the convergence point on the horizon is used as the encoding parameter extracted from the macroblock that satisfies the preset condition. The motion vector is a motion vector of inter-frame prediction in the macroblock.

Images