Image processing device and image processing method

Abstract

To provide a technique for reducing the increase in delay in a case where object recognition processing is performed on a frame to be encoded.SOLUTION: An image processing device performs object recognition processing on a frame, performs encoding processing for the frame, and generates data containing the results of the encoding processing and the results of the recognition processing. If the processing cost of the most recent recognition processing is more than a specified amount, recognition processing is performed only on frames of a B picture.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to encoding techniques. [Background technology]

[0002] In the Annotated Regional SEI (ARSEI) standard of H.265, information such as data indicating the type and position of an object within the field of view can be attached to a stream as metadata.

[0003] The maximum number of objects that can be handled by ARSEI is set to 255 in the specifications, and the coordinates of the top left corner of an object are expressed as 4-byte two-dimensional coordinates, and the width and height of the object are expressed as 2 bytes each. Therefore, the position information of an object (information representing the coordinates of the top left corner of an object, and the width and height of an object) is expressed in a total of 8 bytes.

[0004] In order to attach ARSEI metadata to a stream, it is necessary to carry out object recognition processing on the image before encoding the image. Since advanced recognition processing that identifies the type of object and calculates the position information for each object takes time, delays will occur as the processing load increases, such as when the number of objects increases.

[0005] Furthermore, in a GOP (Group Of Pictures), there is a B picture that refers to past and future frames by bidirectional prediction. If a recognition process is applied to a chronologically subsequent frame that the B picture refers to, the B picture can be encoded after the recognition process and encoding process of that frame are completed, so the delay increases further. For example, as in Patent Document 1, a method is known in which a load of the recognition process is reduced by generating a low-resolution image and applying the recognition process to it. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2000-78563 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in ARSEI, the results of advanced recognition processing, such as object type and position information, are added to the stream as metadata, so the accuracy of the recognition processing for low-resolution images is insufficient. Therefore, the prior art does not provide a solution to the increased delay caused by advanced recognition processing and encoding. The present invention provides a technology for suppressing the increase in delay when object recognition processing is performed on frames to be encoded. [Means for solving the problem]

[0008] One aspect of the present invention comprises a recognition means for performing object recognition processing on a frame, an encoding means for performing encoding processing on the frame, and a generation means for generating data including a result of the encoding processing and a result of the recognition processing, wherein the recognition means performs the recognition processing only on frames of B pictures if the processing cost of the most recent recognition processing is equal to or greater than a specified amount. Effect of the Invention

[0009] According to the present invention, it is possible to suppress an increase in delay when object recognition processing is performed on a frame to be coded. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of a network camera 100. [Diagram 2] 1A and 1B are diagrams for explaining a case in which a delay occurs due to encoding processing. [Diagram 3] 13A and 13B are diagrams for explaining a case in which the delay is further increased when object recognition processing is added before encoding processing. [Figure 4]5 is a flowchart of a generation process of encoded data by the network camera 100. [Diagram 5] 10 is a flowchart showing details of the process in step S402. [Figure 6] 11 is a flowchart showing details of the process in step S501. [Figure 7] 5A to 5C are diagrams for explaining the effects of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [First embodiment] First, an example of the hardware configuration of a network camera 100, which is an example of an image processing device according to this embodiment, will be described with reference to the block diagram of Fig. 1. As shown in Fig. 1, the network camera 100 has an imaging unit 110 and a controller unit 120.

[0013] First, the imaging unit 110 will be described. Light incident from the outside world through an optical lens 111 forms an image on an imaging element 112. The imaging element 112 is a sensor such as a CCD sensor or a CMOS sensor, and converts the light incident through the optical lens 111 into an analog image signal by photoelectric conversion, and outputs the analog image signal to a signal processing circuit 113 at a downstream stage. The signal processing circuit 113 generates a digital image signal by performing various processes including A / D conversion, color conversion processing, noise removal processing, etc. on the analog image signal. Then, the signal processing circuit 113 outputs an image (frame) based on the generated digital image signal to a memory transfer circuit 115 at a downstream stage. The signal processing circuit 113 may perform such an operation continuously, in which case frames are output from the signal processing circuit 113 continuously. On the other hand, the signal processing circuit 113 may perform such an operation periodically or irregularly, in which case frames are output from the signal processing circuit 113 periodically or irregularly.

[0014] The imaging control circuit 114 controls the operation of the imaging element 112 in the same cycle as the image output cycle. When the image accumulation time is longer than the image output cycle, the imaging control circuit 114 controls the signal processing circuit 113 to hold frames in the frame memory of the signal processing circuit 113 during the period when the imaging element 112 cannot output an analog image signal.

[0015] When a frame is output from the signal processing circuit 113 , the memory transfer circuit 115 transfers the frame to the memory 122 in the controller unit 120 .

[0016] Next, a description will be given of the controller unit 120. The CPU 121 executes various processes using computer programs and data stored in the non-volatile memory 124. As a result, the CPU 121 controls the operation of the entire network camera 100, and executes or controls various processes that will be described as processes performed by the network camera 100.

[0017] The memory 122 has an area for storing frames transferred from the memory transfer circuit 115, a work area used when the CPU 121 and the encoding circuit 125 execute various processes, and the like.

[0018] The non-volatile memory 124 stores setting data for the network camera 100, computer programs and data related to the startup of the network camera 100, computer programs and data related to the basic operations of the network camera 100, etc. The non-volatile memory 124 also stores computer programs and data for causing the CPU 121 to execute or control various processes described as processes performed by the network camera 100. The computer programs and data stored in the non-volatile memory 124 are loaded into the memory 122 as appropriate under the control of the CPU 121, and become targets for processing by the CPU 121.

[0019] The encoding circuit 125 performs a recognition process (object recognition process) for recognizing an object included in a frame stored in the memory 122 and collecting information related to the object (such as the type and position of the object), and an encoding process for encoding the frame. In this embodiment, the encoding circuit 125 is described as performing both the object recognition process and the encoding process, but instead of the encoding circuit 125, a circuit for performing the object recognition process and a circuit for encoding a frame may be provided.

[0020] The encoding circuit 125 then generates encoded data including the result of the object recognition process (information related to the object) as metadata and the result of the encoding process as main data. The encoding circuit 125 then outputs the generated encoded data to the external network device 130 via the network I / F 123. The network I / F 123 is an interface for performing data communication with the network device 130.

[0021] The network device 130 is a device for performing data communication between the network camera 100 and the information processing device 140, and is, for example, a network hub. The network device 130 transmits the encoded data output from the network camera 100 via the network I / F 123 to the information processing device 140 via a wired and / or wireless network.

[0022] The information processing device 140 is a computer device such as a PC (personal computer), a tablet terminal device, or a smartphone. For example, the information processing device 140 receives encoded data transmitted from the network camera 100 via the network device 130, decodes the received encoded data, and displays frames and metadata obtained by the decoding on a display device such as a monitor. For example, the information processing device 140 can perform various settings on the network camera 100 and obtain data stored in the non-volatile memory 124 via the network device 130.

[0023] Next, a case where the delay increases due to relatively advanced object recognition processing and encoding processing for each frame will be described. First, a case where a delay occurs due to encoding processing will be described with reference to Fig. 2. In Fig. 2, in order to easily explain a case where a delay occurs due to encoding processing, it is assumed that object recognition processing is not performed on the frame.

[0024] I1, B1, B2, P1, B3, B4, and P2 indicate captured frames, which are arranged in chronological order at the end of the imaging process. That is, the imaging process is completed in the order of I1, B1, B2, P1, B3, B4, and P2. These frames are encoded after the imaging process.

[0025] I1 is a frame (a frame whose picture type is I-picture) that can be encoded / decoded independently in a GOP. B1, B2, B3, and B4 are frames (frames whose picture type is B-picture) that are encoded / decoded with reference to past and future frames. P1 and P2 are frames (frames whose picture type is P-picture) that are encoded / decoded with reference to past frames. In the example of FIG. 2, P-pictures are pictures that are encoded / decoded with reference to past I-pictures, and B-pictures are pictures that are encoded / decoded with reference to past I-pictures and future P-pictures.

[0026] First, I1 is coded, but as mentioned above, I1 is an I-picture, and since an I-picture can be coded independently, it can be coded immediately after the imaging process is completed. The coding of the next B1 requires reference to both I1 and P1, but B1 cannot be coded until the coding of both I1 and P1 is completed, so B1 is coded after the coding of P1 is completed. In this way, when coding a B-picture, a delay due to coding is always generated because future frames are referenced.

[0027] Next, a case where the delay increases further when object recognition processing is added before the encoding processing in order to add the result of the object recognition processing to the encoding processing as metadata will be described with reference to FIG.

[0028] In the example of FIG. 3, after the imaging process, the object recognition process is performed and then the encoding process is performed. I1, B1, B2, P1, B3, B4, and P2 are the same as in FIG. 2. Also, in FIG. 3, the imaging process and the encoding process are performed at a processing speed of 30 fps, and the object recognition process is performed at a processing speed of 10 fps. In this case, since the processing speed of the object recognition process is one-third of the imaging process, in order to maintain the output of the encoding process at 30 fps, the object recognition process can only be applied to one out of three frames that have been imaged. Therefore, in the case of FIG. 3, the object recognition process is applied to I1, P1, and P2. In the object recognition process in FIG. 3, the frames surrounded by the solid line are frames to which the object recognition process is applied, and the frames surrounded by the dotted line are frames to which the object recognition process is not applied and which proceed directly to the encoding process.

[0029] First, when the imaging process of I1 is completed, object recognition process is applied to I1. While the object recognition process is being performed on I1, the imaging process of B1 is completed, but the object recognition process is not applied to B1 and the process proceeds to the encoding process. However, since the encoding of both I1 and P1 must be completed in order to encode B1, encoding of B1 cannot be performed at this point. Therefore, B1 is put into a standby state until the encoding of P1 is completed. The same is true for the following B2. After that, the imaging process of P1 is completed, and since the object recognition process of I1 is completed at this point, object recognition process is applied to P1 before the process proceeds to the encoding process. After the encoding of P1 is completed, encoding of B1 and B2 can finally be performed.

[0030] In this way, when object recognition processing is performed on a group of pictures including B pictures before the encoding processing, the execution time of the object recognition processing is added directly to the delay of the encoding processing, which may increase the delay.

[0031] In this embodiment, if the number of objects recognized in the most recent frame in the object recognition process is equal to or greater than a specified number, the frames to be recognized are limited to only B pictures, and the object recognition process is performed only on those frames to be recognized, thereby preventing the above-mentioned increase in delay.

[0032] The process of generating encoded data by the network camera 100 according to this embodiment will be described with reference to the flowchart of FIG.

[0033] In step S401, the imaging unit 110 performs imaging processing to generate one frame of a captured image, and transfers the generated captured image (frame) to the controller unit 120. The transferred frame is stored in the memory 122 of the controller unit 120.

[0034] In step S402, the encoding circuit 125 in the controller unit 120 performs object recognition processing on the frame transferred from the imaging unit 110 and stored in the memory 122. Details of the processing in step S402 will be described with reference to the flowchart in FIG.

[0035] In step S501, the encoding circuit 125 performs a determination process for determining the type of picture (recognition target picture type) that is to be subjected to object recognition processing. Details of the process in step S501 will be described with reference to the flowchart in FIG.

[0036] In step S601, the encoding circuit 125 judges whether or not the number of objects recognized from a frame in the most recent object recognition process is equal to or greater than a threshold (prescribed number). The threshold is not limited to a value set by a specific setting method, and may be, for example, a predetermined prescribed value or a value set by a user operating the information processing device 140 or the network camera 100.

[0037] If the result of this determination is that the number of objects recognized from the frame in the most recent object recognition process is equal to or greater than the threshold, the process proceeds to step S602. On the other hand, if the number of objects recognized from the frame in the most recent object recognition process is less than the threshold, the process proceeds to step S603.

[0038] In step S602, the encoding circuit 125 sets the recognition target picture type to B picture. That is, the encoding circuit 125 sets a frame corresponding to a B picture as a recognition target frame. In general, in the case of advanced object recognition processing such as recognizing the type of object, the more the number of objects in a frame, the more time it takes to perform the object recognition processing on the frame. Therefore, in this embodiment, when the number of objects recognized from a frame in the most recent object recognition processing is relatively large (above a threshold), it is determined that the object recognition processing on the frame to be subjected to the object recognition processing will take a relatively long time, and the target of the object recognition processing is limited to B picture. This allows the object recognition processing on the B picture to be performed during the waiting time for the encoding processing of the B picture, so that an increase in delay can be prevented.

[0039] In step S603, the encoding circuit 125 determines that performing object recognition processing on all pictures (I pictures, B pictures, and P pictures) will not take enough time to increase delay, and does not limit the recognition target picture type to B pictures. That is, the encoding circuit 125 sets all pictures (I pictures, B pictures, and P pictures) as recognition target picture types. That is, the encoding circuit 125 sets all frames as recognition target frames regardless of picture type.

[0040] Then, the process proceeds to step S502. In step S502, the encoding circuit 125 determines whether or not the object recognition process is executable. For example, if the object recognition process is not being executed for the previous frame, the encoding circuit 125 determines that the object recognition process is executable. On the other hand, if the object recognition process is being executed for the previous frame, the encoding circuit 125 determines that the object recognition process is not executable.

[0041] As a result of this determination, if it is determined that the object recognition process is executable, the process proceeds to step S503, and if it is determined that the object recognition process is not executable, the process proceeds to step S403.

[0042] In step S503, the encoding circuit 125 determines whether or not the current frame (the frame to be encoded) corresponds to a frame to be recognized (whether or not the current frame is a frame of the picture type to be recognized).

[0043] If the result of this determination is that the current frame corresponds to a frame to be recognized (the current frame is a frame of the picture type to be recognized), the process proceeds to step S504. On the other hand, if the current frame does not correspond to a frame to be recognized (the current frame is not a frame of the picture type to be recognized), the process proceeds to step S403.

[0044] In step S504, the encoding circuit 125 performs an object recognition process on the current frame. This object recognition process is a relatively advanced recognition process, for example, a process of collecting various information related to the object, such as the type of the object included in the frame, the position of the object, and the like.

[0045] Since the object recognition process is executed before the encoding process, the picture type of the current frame has not been determined at the time of the object recognition process. However, it is easy to determine which picture type the current frame will be from the order of picture types within the GOP.

[0046] In step S403, the encoding circuit 125 performs an encoding process to encode the frame transferred from the imaging unit 110 and stored in the memory 122. Then, the encoding circuit 125 generates encoded data including the result of the object recognition process in step S402 as metadata and including the result of the encoding process in step S403 as main data.

[0047] Next, the effect of this embodiment will be described with reference to Fig. 7. Here, it is assumed that a B picture is set as a recognition target frame. In this case, when the imaging process of I1 is completed, since the picture type of I1 is an I picture, I1 is determined not to be a recognition target frame, and therefore object recognition process is not performed on I1, and encoding process is performed.

[0048] Next, when the imaging process for B1 is completed, since the picture type of B1 is a B picture, B1 is determined to be a frame to be recognized, and therefore object recognition process is executed for B1.

[0049] After that, the imaging process for B2 ends, but although the picture type of B2 is a B picture and corresponds to the frame to be recognized, it is determined that the object recognition process for B2 cannot be performed because the object recognition process for B1 is currently being performed. Therefore, the object recognition process for B2 is not performed.

[0050] Next, the imaging process for the P1 frame is completed. However, since the picture type of P1 is a P picture, it is determined that P1 does not correspond to a frame to be recognized. Therefore, object recognition process is not performed on P1, and encoding process is performed thereon.

[0051] After that, when the object recognition process for B1 is completed, the encoding of I1 and P1 is completed, so the encoding process for B1 can be executed immediately after the object recognition process is completed. Note that when the encoding of I1 and P1 is completed, the encoding of B2 becomes possible, so the encoding process can be executed for B2 as well.

[0052] Thus, according to this embodiment, object recognition processing can be performed during the waiting time for the encoding processing, thereby preventing increased delay that would occur by performing advanced object recognition processing before the encoding processing.

[0053] [Second embodiment] In the above step S601, it may be determined whether the time required for the most recent object recognition process is equal to or greater than a threshold (a specified time or greater). In this case, if the time required for the most recent object recognition process is equal to or greater than the threshold, the process proceeds to step S602, and if the time required for the most recent object recognition process is less than the threshold, the process proceeds to step S603.

[0054] In addition, in the above step S601, it may be determined whether the average time required for the object recognition process in the most recent specified number of frames is equal to or greater than a threshold (specified time or greater). In this case, if the average time is equal to or greater than the threshold, the process proceeds to step S602, and if the average time is less than the threshold, the process proceeds to step S603.

[0055] The specified time may be a predetermined time, or the time available for object recognition processing may be calculated from the frame rate of the image capture processing, and the calculated time may be set as the specified time. For example, when the image capture processing has a frame rate of 30 fps, the processing time per frame is 33 milliseconds or less, and the time available for object recognition processing is calculated by subtracting the time required for encoding processing and other processing from the frame rate.

[0056] In this way, it is determined whether the processing cost of the most recent recognition processing is equal to or greater than a specified amount, and if the processing cost of the most recent recognition processing is equal to or greater than a specified amount, various forms are possible for performing object recognition processing only on frames of B pictures, and are not limited to a specific form.

[0057] In addition, when the set frame rate exceeds a predetermined reference frame rate, only B pictures may be set as frames to be recognized, or whether or not to set only B pictures as frames to be recognized may be determined according to a user operation.

[0058] 1 shows the network camera 100 in which the imaging unit 110 and the controller unit 120 are integrated, the imaging unit 110 and the controller unit 120 may be separate devices. The controller unit 120 may be incorporated in the information processing device 140.

[0059] Furthermore, the destination of the encoded data is not limited to the information processing device 140. For example, the network camera 100 may transmit the generated encoded data to a device on the network, such as a server device, and store the encoded data in the server device, or may transmit the encoded data to a television device as broadcast data.

[0060] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0061] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0062] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0063] The invention of this specification includes the following image processing device, image processing method, and computer program.

[0064] (Item 1) A recognition means for performing object recognition processing on the frame; An encoding means for performing a frame encoding process; a generating means for generating data including a result of the encoding process and a result of the recognition process; Equipped with The recognition means performs the recognition process only on frames of B pictures if the processing cost of the most recent recognition process is equal to or greater than a specified amount. 13. An image processing device comprising:

[0065] (Item 2) 2. The image processing device according to item 1, wherein the recognition means performs the recognition process only on frames of B pictures if the number of objects in a frame recognized in the most recent recognition process is equal to or greater than a specified number.

[0066] (Item 3) 2. The image processing device according to item 1, wherein the recognition means performs the recognition process only on frames of B pictures when the time required for the most recent recognition process is equal to or longer than a specified time.

[0067] (Item 4) 2. The image processing device according to item 1, characterized in that the recognition means performs the recognition processing only on frames of B pictures when the average time required for the recognition processing of the most recent specified number of frames is equal to or longer than the specified time.

[0068] (Item 5) moreover, 5. The image processing device according to any one of items 1 to 4, further comprising an output unit that outputs the data generated by the generation unit.

[0069] (Item 6) Further, the imaging means is provided, The recognition means performs the recognition process on the frame captured by the imaging means, The encoding means performs encoding processing of the frames captured by the imaging means. 6. The image processing device according to any one of items 1 to 5,

[0070] (Item 7) 7. The image processing device according to item 6, wherein the image processing device is a network camera.

[0071] (Item 8) An image processing method performed by an image processing device, comprising: a recognition step in which a recognition means of the image processing device performs object recognition processing on the frame; an encoding step in which an encoding means of the image processing device performs encoding processing of a frame; a generating step in which a generating means of the image processing device generates data including a result of the encoding process and a result of the recognition process; Equipped with In the recognition step, if the processing cost of the most recent recognition process is equal to or greater than a specified amount, the recognition process is performed only on frames of B pictures. 13. An image processing method comprising:

[0072] (Item 9) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 7.

[0073] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0074] 100: Network camera 110: Imaging unit 111: Optical lens 112: Imaging element 113: Signal processing circuit 114: Imaging control circuit 115: Memory transfer circuit 120: Controller unit 121: CPU 122: Memory 123: Network I / F 124: Non-volatile memory 125: Encoding circuit 130: Network device 140: Information processing device

Claims

1. a recognition means for performing object recognition processing on the frame; encoding means for performing frame encoding processing; a generating means for generating data including a result of the encoding process and a result of the recognition process; Equipped with The recognition means performs the recognition process only on frames of B pictures if the processing cost of the most recent recognition process is equal to or greater than a specified amount.

1. An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein said recognition means performs said recognition processing only on frames of B pictures if the number of objects in a frame recognized in the most recent recognition processing is equal to or greater than a specified number.

3. 2. The image processing device according to claim 1, wherein said recognition means performs said recognition process only on frames of B pictures when the time required for the most recent recognition process is equal to or longer than a specified time.

4. 2. The image processing device according to claim 1, wherein said recognition means performs said recognition processing only on frames of B pictures when an average time required for the recognition processing of a specified number of most recent frames is equal to or longer than a specified time.

5. The image processing device described in Claim 1, characterized in that the recognition means performs the recognition processing only on frames of B pictures when the set frame rate exceeds a predetermined standard frame rate.

6. The image processing device described in Claim 1, characterized in that the recognition means performs the recognition processing for all frames if the processing cost of the most recent recognition processing is less than a specified amount.

7. moreover, 2. The image processing apparatus according to claim 1, further comprising an output unit that outputs the data generated by the generation unit.

8. Further, the imaging means is provided, the recognition means performs the recognition processing on the frame captured by the imaging means, The encoding means performs encoding processing of the frames captured by the imaging means.

2. The image processing device according to claim 1, wherein:

9. 9. The image processing device according to claim 8, wherein the image processing device is a network camera.

10. An image processing method performed by an image processing device, a recognition step in which recognition means of the image processing device performs object recognition processing on the frame; an encoding step in which encoding means of the image processing device performs encoding processing of a frame; a generation step in which a generation means of the image processing device generates data including a result of the encoding process and a result of the recognition process; Equipped with In the recognition step, if the processing cost of the most recent recognition process is equal to or greater than a specified amount, the recognition process is performed only on frames of B pictures. An image processing method comprising:

11. A program for executing the image processing method of the image processing device described in claim 10.

Images