Imaging device, information processing method, and program

Abstract

To suppress increase of information transmission amount in transmission of metadata relating to objects detected from an image.SOLUTION: A method according to the present invention has a step of detecting an object included in an image captured by an imaging means, a step of generating metadata compliant with an ARSEI and including positional information of the objected detected from the image, a step of transmitting, to an external device, an encoded data generated by encoding the image and distribution data including the meta data relating to the image, and a step of, if a change amount of a parameter of a first object detected from a second image captured after the first image relative to the parameter of the first object detected from the first image is smaller than a threshold value, generating, as the meta data relating to the second image, a meta data which does not include the positional information of the first object in the second image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing method. [Background technology]

[0002] High Efficiency Video Coding (HEVC) is known as a video coding method (see Non-Patent Document 1). In addition, ARSEI (Annotated Regions SEI) in the HEVC standard makes it possible to distribute information indicating the position of an object in an image and label information as metadata.

[0003] Furthermore, as a conventional technology for distributing position information of objects in an image as metadata, Patent Document 1 discloses a technology in which a camera transmits metadata including position information of objects detected from an image to a client. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-9134 [Non-patent literature]

[0005] [Non-Patent Document 1] ITU-T H.265(11 / 2019)High efficiency video coding Summary of the Invention [Problem to be solved by the invention]

[0006] Here, when transmitting position information of an object detected from an image to an external device, one possible method is to transmit position information of the object detected for each image of each frame that constitutes the video to the external device. However, even when it is not necessary to transmit position information for a certain object multiple times, such as when the position of the object does not change within a series of images, the position information may be transmitted, resulting in an increase in the amount of information transmitted.

[0007] Therefore, an object of the present invention is to suppress an increase in the amount of information to be transmitted when transmitting metadata related to objects detected from an image. [Means for solving the problem]

[0008] In order to solve the above problem, for example, an imaging device according to the present invention has the following configuration: That is, the imaging device has a detection means for detecting an object included in an image captured by an imaging means, a generation means for generating metadata compliant with ARSEI including position information of the object detected from the image by the detection means, and a transmission means for transmitting to an external device encoded data generated by encoding the image and distribution data including the metadata related to the image, wherein the generation means generates, as metadata related to the second image, metadata that does not include position information of the first object in the second image when an amount of change in the parameter of the first object detected from the first image relative to the parameter of the first object detected from the second image is less than a threshold value. When a plurality of objects are detected from the image, the generating means identifies a change amount of a parameter for each object, and selects an object whose position information is to be included in the metadata based on the change amount identified for each object and a set number that is an upper limit of the number of objects to be updated. An imaging device characterized by: [Effects of the Invention]

[0009] According to the present invention, when transmitting metadata related to an object detected from an image, an increase in the amount of information to be transmitted can be suppressed. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a system configuration. [Figure 2] FIG. 2 is a diagram illustrating functional blocks of the imaging device. [Figure 3] FIG. 1 is a diagram illustrating the data structure of ARSEI. [Figure 4] FIG. 10 is a diagram illustrating a metadata generation process. [Figure 5] 10 is a flowchart showing the flow of a metadata generation process. [Figure 6] FIG. 10 is a diagram illustrating a metadata generation process. [Figure 7] 10 is a flowchart showing the flow of a metadata generation process. [Figure 8] FIG. 2 illustrates an example of a hardware configuration of each device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the configurations shown in the drawings.

[0012] (Embodiment 1) 1 is a diagram showing the system configuration of this embodiment. The system of this embodiment includes an image capture device 100, a client device 101, a display 103, and a network .

[0013] The imaging device 100 and the client device 101 are connected to each other via a network 102. The network 102 is realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark).

[0014] The network 102 may be realized by the Internet, a wired local area network (LAN), a wireless LAN, a wide area network (WAN), or the like.

[0015] The imaging device 100 is a device that captures images and also functions as an image processing device that processes images. The imaging device 100 transmits distribution data including coded data obtained by encoding the captured images to an external device such as a client device 101 via a network 102. The client device 101 is, for example, an information processing device such as a personal computer in which a program for realizing the processing functions described below is installed.

[0016] The display 103 is configured with an LCD (Liquid Crystal Display) or the like, and displays a decoded image obtained by the client device 102 decoding encoded data included in the distribution data transmitted from the imaging device 100. The display 103 is connected to the client device 101 via a display cable that complies with a communication standard such as HDMI (High Definition Multimedia Interface), a registered trademark. The display 103 and the client device 101 may be provided in a single housing.

[0017] The following description will be made with reference to the functional blocks shown in Fig. 2. Fig. 2 shows the functional blocks of the image capture device 100 according to this embodiment. Note that the functions of the functional blocks shown in Fig. 2 are realized by a CPU (Central Processing Unit) 800 of the image capture device 100 executing a computer program stored in a ROM (Read Only Memory) 820 of the image capture device 100, which will be described later with reference to Fig. 8, for example.

[0018] The imaging unit 201 captures a subject image using an imaging element such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor, and generates an electrical signal by photoelectrically converting the captured image. The imaging unit 201 then converts the photoelectrically converted electrical signal into a digital signal to generate an image.

[0019] The encoding processing unit 202 performs encoding processing on the image captured by the imaging unit 201, and generates encoded data of the image. Note that the encoding processing unit 202 in this embodiment uses, for example, HEVC as the encoding processing for the image.

[0020] The detection unit 203 detects an object included in the image input from the imaging unit 201. In this embodiment, the object to be detected will be described as a person. The detection unit 203 detects a person included in the image by performing processing such as pattern matching using a matching pattern (dictionary), for example. When detecting a person from an image, the detection unit 203 may detect a person from the image using multiple matching patterns, such as a matching pattern when the person is facing forward and a matching pattern when the person is facing sideways. In this way, by performing detection processing using multiple matching patterns, improvement in detection accuracy can be expected.

[0021] In this embodiment, a person is detected as an object to be detected from an image, but the object is not limited to a person and may be another object such as a car. Also, although the detection unit 203 in this embodiment uses pattern matching processing as a method for detecting an object from an image, other conventional object detection techniques may be used to detect an object from an image.

[0022] The metadata generation unit 204 generates metadata related to the image in accordance with the detection result of the object detected from the image by the detection unit 203. Note that although the metadata generation unit 204 in this embodiment generates metadata that complies with ARSEI, it may also generate metadata that complies with other formats.

[0023] The distribution data generation unit 205 generates distribution data including encoded data obtained by encoding processing of the image captured by the imaging unit 201 and metadata generated by the metadata generation unit 204 based on the detection results detected from the image.

[0024] The transmission unit 206 transmits the distribution data generated by the distribution data generation unit 205 to an external device via the network 102.

[0025] Fig. 3 shows the ARSEI data used in this embodiment. Hereinafter, the generation of metadata in ARSEI by the metadata generation unit 204 in this embodiment will be described with reference to Fig. 3.

[0026] Data 300 shown in Figure 3 is a pseudocode representation of the ARSEI data structure. In data 300, the gray columns are the parts related to data structure control. In addition, in data 300, the white columns are the parts where the actual data is stored, if data to be stored exists, according to the gray data structure control. ARSEI includes ar_label_idx[i] (301) that specifies ar_label (302), which can store up to 255 bytes of information (such as text), and ar_object_idx[i] (304), which identifies objects in the image. In addition, up to 256 ar_label_idx[i] and ar_object_idx[i] can be registered, and it is possible to determine which label to assign to an object when the object is updated.

[0027] Here, ar_num_object_updates (303) in ARSEI data 300 associated with a certain image indicates the number of objects in the certain image whose information has been updated. For example, if three objects are detected by the detection unit 203 in a captured image and information on two of the objects is to be updated, ar_num_object_updates (303) in the data 300 for the certain image will be 2. If ar_num_object_updates (303) is non-zero (i.e., 1 or greater), an index (any value between 0 and 255) identifying the object whose position information is to be updated is entered in ar_object_idx[i] (304). For example, assume that three objects are detected in the image of the previous frame, and each object is assigned an index "0," "1," and "2" as ar_object_idx (304), and only the object with index "2" in the image of the current frame needs its position information updated. Here, in the data 300 generated for the image of the current frame, “2” is stored in ar_object_idx[0] (304), and ar_bounding_box_update_flag (305) is set to 1. Then, position information of the top left vertex of the bounding box of the object with index “2” in the image of the current frame is stored in ar_bounding_box_top (307) and ar_bounding_box_left (308), information on the width of the bounding box is stored in ar_bounding_box_width (309), and information on the height of the bounding box is stored in ar_bounding_box_height (310).

[0028] Furthermore, by setting the ar_object_cancel_flag (305) for a certain numerical value of ar_object_idx[i] (304) to 1, information about the object of that ar_object_idx can be deleted.

[0029] The metadata generating unit 204 stores information such as ar_object_idx (304) in the data 300 according to the result of object detection for the image, and generates a NAL unit corresponding to the data 300 as metadata related to the image.

[0030] The specifications for ARSEI stipulate that the maximum number of updatable objects handled by ARSEI is 255. In ARSEI, the coordinates of the upper-left vertex of an object's bounding box are expressed as 4-byte two-dimensional coordinates, and the width and height of the bounding box are each expressed as 2 bytes, resulting in a total of 8 bytes for the object's position information. If the position information for each of the 255 objects in an image is updated at 30 FPS, 8 bytes × 255 × 30 ≒ 500 kbps of information would be required just to update the position information. Updating the position information (ar_bounding_box_top, ar_bounding_box_left) and size information (ar_bounding_box_width, ar_bounding_box_height) for all objects in each image constituting a video would increase the amount of information transmitted, potentially constricting network bandwidth. Therefore, the metadata generation unit 204 in this embodiment determines the amount of change in the parameters of a first object detected in a second image captured after the first image, relative to the parameters of the first object detected in the first image. The metadata generation unit 204 then compares the identified amount of change with a predetermined threshold, and if the amount of change is less than the predetermined threshold, generates metadata relating to the second image that does not include position information and size information of the first object. On the other hand, if the amount of change is equal to or greater than the predetermined threshold, the metadata generation unit 204 generates metadata relating to the second image that includes position information and size information of the first object.

[0031] Now, with reference to FIG. 4, the process of generating metadata by the imaging device 100 in this embodiment will be described in more detail. FIG. 4 shows a series of images 400 to 402 captured by the imaging unit 201. In time series, image 400 is the Nth frame (N is an integer), image 401 corresponds to the N+1th frame as the image captured next, and image 402 corresponds to the N+2th frame as the image captured after image 401. Also, as shown in FIG. 4, object 403, which is the same object, is detected by the detection unit 203 from each of images 400 to 402. Here, in image 400 of the Nth frame, a circumscribing rectangle 404 indicates the bounding box of object 403 detected from image 400. A dashed-line frame 405 of image 400 indicates the circumscribing rectangle of object 403 in a frame prior to image 400, in which position information for object 403 was last transmitted in accordance with ARSEI. That is, the dashed frame 405 in the image 400 indicates the position and size of the object 403 in the image 400 when the last ARSEI was transmitted for the object 403.

[0032] The metadata generation unit 204 also identifies, as the amount of change, the size of a non-overlapping area 406 between the area of ​​the dashed-line frame 405 and the area of ​​the circumscribing rectangle 404 in the image 400, which is the current frame to be processed. The metadata generation unit 204 in this embodiment then determines, based on the size of the non-overlapping area, whether to include position information and size information of the object 403 in the metadata related to the image 400. For example, the metadata generation unit 204 compares the size of the non-overlapping area of ​​the object 403 with a predetermined threshold, and if it determines that the size of the non-overlapping area is less than the predetermined threshold, does not include the position information and size information of the object 403 in the metadata. In this case, the metadata generation unit 204 does not update the ar_bounding_box_top (307), ar_bounding_box_left (308), ar_bounding_box_width (309), and ar_bounding_box_height (310) of the object 403 in the data 300 related to the image 400. On the other hand, if the metadata generation unit 204 determines that the size of the non-overlapping region is equal to or larger than the predetermined threshold, it includes in the metadata the position information and size information of the object 403. In this case, the metadata generation unit 204 updates ar_bounding_box_top (307), ar_bounding_box_left (308), ar_bounding_box_width (309), and ar_bounding_box_height (310) in the data 300 related to the image 400 according to the position information and size of the object 403 in the image 400.

[0033] Although the example in which the size of the non-overlapping region is compared with a predetermined threshold and whether to include position information and size information in the metadata is determined based on the comparison result has been described, the present invention is not limited to this. For example, the metadata generation unit 204 may identify the ratio of the size of the non-overlapping region to the region defined by the circumscribing rectangle 404 of the image 400 (the region inside the circumscribing rectangle 404) and compare the ratio with a predetermined threshold. If the ratio is less than the predetermined threshold, the metadata generation unit 204 does not update ar_bounding_box_top (307), ar_bounding_box_left (308), ar_bounding_box_width (309), and ar_bounding_box_height (310). If the ratio is equal to or greater than the predetermined threshold, the metadata generation unit 204 updates ar_bounding_box_top (307), ar_bounding_box_left (308), ar_bounding_box_width (309), and ar_bounding_box_height (310). In the example shown in Figure 4, the size of the non-overlapping area 406 of the object 403 in the image 400 is assumed to be less than a predetermined threshold, and the ar_bounding_box_top (307), ar_bounding_box_left (308), ar_bounding_box_width (309), and ar_bounding_box_height (310) for the object 403 are not updated.

[0034] Next, a circumscribing rectangle 407 in image 401 corresponding to the (N+1)th frame indicates the bounding box of object 403 detected in image 401. Furthermore, as described above, a dashed-line frame 405 indicates the circumscribing rectangle of object 403 in the frame in which position information for object 403 was last transmitted in accordance with ARSEI. Since position information for object 403 in image 400 was not transmitted in accordance with ARSEI, the dashed-line frame 405 is provided at the same position in both images 400 and 401. The metadata generation unit 204 identifies a non-overlapping region 408 between the region of dashed-line frame 405 and the region of circumscribing rectangle 407 in image 401, which is the currently processed frame. Based on the size of the non-overlapping region 408, the metadata generation unit 204 determines whether to include position information and size information for object 403 as metadata related to image 401. In the example shown in FIG. 4, the size of the non-overlapping region 408 for object 403 in image 401 is equal to or greater than a predetermined threshold. Therefore, the metadata generating unit 204 updates the ar_bounding_box_top (307), ar_bounding_box_left (308), ar_bounding_box_width (309), and ar_bounding_box_height (310) as metadata for the image 401 according to the position information and size information of the object 403 in the image 401.

[0035] Next, in image 402, which is the (N+2)th frame, a circumscribing rectangle 409 indicates the bounding box of object 403 detected in image 402. A dashed-line frame 410 indicates the circumscribing rectangle of object 403 when information conforming to ARSEI was last transmitted for object 403 in image 402. That is, the position of dashed-line frame 410 corresponds to the position of the circumscribing rectangle of object 403 in image 401 in the example shown in FIG. 4 . The metadata generation unit 204 identifies a non-overlapping region 411 between the region of dashed-line frame 410 and the region of circumscribing rectangle 409 in image 402, which is the currently processed frame. Then, based on the size of non-overlapping region 411, the metadata generation unit 204 determines whether to include position information and size information of object 403 as metadata related to image 402. In the example shown in FIG. 4 , it is assumed that the size of non-overlapping region 411 for object 403 in image 402 is less than a predetermined threshold. Therefore, the metadata generating unit 204 does not update the ar_bounding_box_top (307), ar_bounding_box_left (308), ar_bounding_box_width (309), and ar_bounding_box_height (310) of the object 403 in the image 402 as metadata for the image 402.

[0036] As described above, the metadata generation unit 204 determines the circumscribing rectangle of the first object in the first image when the position information was last transmitted as the parameters of the first object in the first image. The metadata generation unit 204 also determines the circumscribing rectangle of the first object detected in the second image captured after the first image as the parameters of the first object in the second image. The metadata generation unit 204 then determines the size of the non-overlapping region as the amount of change in the parameters of the first object detected in the second image captured after the first image relative to the parameters of the first object in the first image. The metadata generation unit 204 then determines whether to include the position information of the certain object in the metadata of the image currently being processed, based on the comparison result between the size of the non-overlapping region and a predetermined threshold. In this way, the metadata generation unit 204 determines whether to include the position information and size information of the object in the metadata of the image being processed, based on the amount of change in the parameters since the last time the ARSEI position information and size information were transmitted. This makes it possible to reduce the amount of metadata to be transmitted compared to uniformly transmitting metadata including the position information and size information of a certain object for each of all images.

[0037] Next, a process for generating metadata by the imaging device 100 in this embodiment will be described with reference to Fig. 5. Note that the process of the flow shown in Fig. 5 is executed by the functional blocks shown in Fig. 2 which are realized by the CPU 800 of the imaging device 100 executing a computer program stored in the ROM 820 of the imaging device 100, for example.

[0038] First, in S501, the detection unit 203 acquires an image captured by the imaging unit 201 and that is to be currently processed. Next, in S502, the detection unit 203 executes processing to detect objects in the image acquired in S501. Next, in S503, the metadata generation unit 204 initializes j to zero, and if j<number of objects is satisfied, executes a series of processing steps from S504 to S507, incrementing j by 1 each time the processing steps from S504 to S507 are executed. Note that the number of objects here indicates the number of objects detected from the image currently being processed. Therefore, the series of processing steps from S504 to S507 are executed for each object in the image currently being processed.

[0039] In S504, the metadata generation unit 204 executes the following process. That is, a non-overlapping area is identified from the circumscribing rectangular area of ​​an object in the image currently being processed (hereinafter referred to as the target object) and the circumscribing rectangular area of ​​the target object in the image when information about the target object was last included in the metadata. Here, if the image currently being processed is image 400 shown in FIG. 4, in S504 the metadata generation unit 204 identifies non-overlapping area 406.

[0040] In S505, the metadata generation unit 204 determines whether to update the position information and size information according to the size of the non-overlapping area of ​​the target object identified in S504 on the image currently being processed. As described with reference to FIG. 4, the metadata generation unit 204 executes, for example, the following process. That is, if the size of the non-overlapping area of ​​the target object is equal to or greater than a predetermined threshold, it determines to update the position information and size information (Yes in S505). On the other hand, if the size of the non-overlapping area is less than the predetermined threshold, it determines not to update the position information and size information (No in S505). If it is determined not to update the position information and size information (No in S505), the process proceeds to S503, where the metadata generation unit 204 increments j by 1, and executes the processes of S504 to S507 for the next target object. If it is determined not to update the position information and size information (Yes in S505), the process proceeds to S506. In S506, the metadata generation unit 204 stores the position information of the target object in the image currently being processed in ar_bounding_box_top, (307) ar_bounding_box_left (308) as data 300 of the image, and stores the size information of the target object (width and height of the bounding box) in ar_bounding_box_width (309) and ar_bounding_box_height (310).

[0041] Next, in S507, the metadata generation unit 204 stores the position information and size information of the target object in the image currently being processed, and transitions to S503. In S503, the metadata generation unit 204 increments j by 1, and executes the processes of S504 to S507 for the next target object. Note that if the condition "j<number of objects" is not met in S503, in other words, if the processes of S504 to S507 have been executed for all objects in the image currently being processed, transitions to S508.

[0042] In S508, the metadata generating unit 204 generates, as metadata related to the image, NAL units corresponding to the data 300 storing information about objects for the image currently being processed.

[0043] In S509, the distribution data generation unit 205 generates distribution data including metadata generated for the image currently being processed and encoded data generated by encoding the image. For example, the distribution data generation unit 205 generates distribution data by storing the metadata generated for the image in the header portion of the encoded data for the image. Next, in S510, the transmission unit 206 outputs the distribution data generated in S509 to an external device. Next, in S511, if an instruction to end has been given by the user (Yes in S511), the processing of the flow shown in FIG. 5 ends; if an instruction to end has not been given (No in S510), the processing returns to S501, and the detection unit 203 acquires the image to be processed next.

[0044] In the above description, the size of the non-overlapping area is used as the amount of change in the parameters of the first object in the second image captured after the first image relative to the parameters of the first object in the first image. However, this is not limiting, and the amount of change in position information may also be used. Specifically, the metadata generation unit 204 identifies position information (ar_bounding_box_top, ar_bounding_box_left) of the upper left vertex of the bounding box of the first object in the first image as the parameters of the first object in the first image for which position information and size information were last transmitted. Similarly, the metadata generation unit 204 identifies position information (ar_bounding_box_top, ar_bounding_box_left) of the upper left vertex of the bounding box of the first object in the second image as the parameters of the first object in the second image captured after the first image. Then, the metadata generation unit 204 identifies the following information as the amount of change in the parameters of the first object in the second image relative to the parameters of the first object in the first image. That is, the amount of change between the position of the upper left vertex of the bounding box of the first object in the first image and the position of the upper left vertex of the bounding box of the first object in the second image is identified. The metadata generation unit 204 compares the identified amount of change in position with a predetermined threshold, and if the amount of change is less than the predetermined threshold, does not include position information and size information of the first object in the metadata for the second image. On the other hand, if the amount of change is equal to or greater than the predetermined threshold, the metadata generation unit 204 includes position information and size information of the first object in the metadata for the second image. In this way, the amount of change in the position of the first object may be used as the amount of change in the parameters of the first object in the second image relative to the parameters of the first object in the first image.

[0045] As described above, the metadata generation unit 204 in this embodiment identifies the amount of change in the parameters of a first object detected in a second image captured after the first image, relative to the parameters of the first object detected in the first image. The metadata generation unit 204 then compares the identified amount of change with a predetermined threshold and, based on the comparison result, determines whether to include position information and size information of the first object in the metadata related to the second image. In this way, if the amount of change in the parameters of a certain object since the last time metadata including position information and size information was transmitted is small, the latest position information and size information are not included in the metadata. This makes it possible to prevent an increase in the amount of metadata transmitted to an external device.

[0046] (Embodiment 2) The imaging device 100 in the second embodiment identifies the amount of change in parameters for each of a plurality of objects from an image, and selects an object whose metadata includes position information and size information according to the amount of change in the parameters of each object. The processing of the imaging device 100 in this embodiment will be described below with reference to Figures 6 and 7. Differences from the first embodiment will be mainly described, and components and processing that are the same as or equivalent to those in the first embodiment will be denoted by the same reference numerals, and redundant description will be omitted.

[0047] The processing of the imaging device 100 in this embodiment will now be described with reference to FIG. 6. Image 600 in FIG. 6 is the image currently being processed, and four objects 601 to 604 have been detected by the detection unit 203. For each of the four objects detected in image 600, the metadata generation unit 204 determines the amount of change in parameters from the frame in which position information and size information were last sent. In this embodiment, the metadata generation unit 204 determines the size of the non-overlapping area as the amount of change in parameters, as in the description of FIG. 4. The metadata generation unit 204 in this embodiment then selects objects in descending order of non-overlapping area size, within a range that does not exceed a preset number set as the number of object updates. The preset number, which is the upper limit of the number of objects to be updated in one frame, may be specified by the user or may be set according to the network bandwidth. Here, for example, it is assumed that the preset number is set to "2." The metadata generation unit 204 selects objects in descending order of non-overlapping area size from among objects whose non-overlapping area size exceeds a predetermined threshold. In the example shown in FIG. 6, the metadata generation unit 204 performs the following processing. That is, among objects 601 to 603, which are objects having non-overlapping area sizes exceeding a predetermined threshold, objects 601 and 603 are selected in descending order of non-overlapping area size, within a range not exceeding the set number "2." Then, the metadata generation unit 204 includes position information and size information of objects 601 and 603 in image 600 in the metadata for image 600, but does not include position information and size information of other objects (objects 602 and 604) in the metadata. That is, the metadata generation unit 204 in this embodiment selects a number of objects equal to or less than the set number according to the amount of change in the parameters of the objects in the image, and includes position information and size information of only the selected objects in the metadata. By imposing a limit on the number of objects for which position information and size information are to be included in the metadata in this way, it is possible to prevent an increase in the amount of metadata information transmitted to an external device.

[0048] Next, a process for generating metadata by the imaging device 100 in this embodiment will be described with reference to Fig. 7. Note that the process of the flow shown in Fig. 7 is executed by the functional blocks shown in Fig. 2 which are realized by the CPU 800 of the imaging device 100 executing a computer program stored in the ROM 820 of the imaging device 100, for example.

[0049] First, in S701, the detection unit 203 acquires an image captured by the imaging unit 201 and that is currently being processed. Next, in S702, the detection unit 203 executes processing to detect objects in the image acquired in S701. Next, in S703, the metadata generation unit 204 executes the following processing to identify non-overlapping areas for each object in the image currently being processed. That is, for each object, the non-overlapping area of ​​the object is identified from the circumscribing rectangular area of ​​the object in the image when information about the object was last included in the metadata and the circumscribing rectangular area of ​​the object in the image currently being processed. Here, if the image currently being processed is image 600 shown in FIG. 6, in S703 the metadata generation unit 204 identifies non-overlapping areas 613-616 for each of objects 601-604.

[0050] In S704, the metadata generation unit 204 updates the objects of the image to be currently processed, whose position information and size information are to be updated, in accordance with the size of each non-overlapping area of ​​each object on the image and the set number. For example, the metadata generation unit 204 selects objects in descending order of non-overlapping area size from among the objects in the image whose non-overlapping area size is equal to or larger than a predetermined threshold, as long as the number of objects to be selected does not exceed the set number. However, without considering the predetermined threshold, the metadata generation unit 204 may select objects in descending order of non-overlapping area size as long as the number of objects to be selected does not exceed the set number.

[0051] In S705, the metadata generation unit 204 stores the position information of each object selected in S704 in the image in ar_bounding_box_top, (307) ar_bounding_box_left (308) as the data 300 of the image currently being processed, and stores the size information of the object (width and height of the bounding box) in ar_bounding_box_width (309) and ar_bounding_box_height (310).

[0052] Next, in S706, the metadata generation unit 204 stores the position information and size information of the object whose position information and size information have been updated. In S707, the metadata generation unit 204 generates, as metadata related to the image, a NAL unit corresponding to data 300 storing object information for the image currently being processed. In S708, the distribution data generation unit 205 generates distribution data including metadata generated for the image currently being processed and coded data generated by coding the image. For example, the distribution data generation unit 205 generates distribution data by storing the metadata generated for the image in the header portion of the coded data for the image. Next, in S709, the transmission unit 206 outputs the distribution data generated in S708 to an external device. Next, in S710, if an end instruction has been given by the user (Yes in S710), the processing flow shown in FIG. 7 ends. If an end instruction has not been given (No in S710), the processing returns to S701, where the detection unit 203 acquires the image to be next processed.

[0053] In the above description, the size of the non-overlapping area is used as the amount of change in the parameters of a certain object in a second image captured after the first image, relative to the parameters of the object in the first image, which is the frame when position information was last transmitted. However, as in the first embodiment, the amount of change in the object may also be the amount of change in the position of the upper left vertex of the bounding box of the object.

[0054] As described above, the metadata generation unit 204 in this embodiment identifies the amount of change in parameters of a first object detected in a second image captured after the first image, relative to the parameters of the first object detected in the first image. The metadata generation unit 204 selects objects for which position information and size information are to be updated, based on the identified amount of change and a set number that is the upper limit of the number of object updates. Then, the metadata generation unit 204 includes position information and size information for the selected objects in the metadata. By limiting the number of objects for which metadata is to be updated in this way, it is possible to suppress an increase in the amount of metadata information transmitted to an external device.

[0055] (Other embodiments) Next, the hardware configuration of the imaging device 100 for realizing each function of each embodiment will be described with reference to Fig. 8. Note that, although the hardware configuration of the imaging device 100 will be described in the following explanation, it is assumed that the imaging device 100 is also realized by a similar hardware configuration.

[0056] The imaging device 100 in this embodiment includes a CPU 800 , a RAM 810 , a ROM 820 , an HDD 830 , and an I / F 840 .

[0057] The CPU 800 is a central processing unit that controls the imaging device 100. The RAM 810 temporarily stores computer programs executed by the CPU 800. The RAM 810 also provides a work area used when the CPU 800 executes processing. The RAM 810 also functions as, for example, a frame memory or a buffer memory.

[0058] The ROM 820 stores programs and the like that are used by the CPU 800 to control the information processing device 200. The HDD 830 is a storage device that records image data and the like.

[0059] The I / F 840 communicates with external devices via the network 102 in accordance with TCP / IP, HTTP, or the like.

[0060] Although the above-described embodiments have been described with reference to examples in which the CPU 800 executes the processing, at least a part of the processing by the CPU 800 may be executed by dedicated hardware. For example, the processing of reading program code from the ROM 820 and loading it into the RAM 810 may be executed by a DMA (Direct Memory Access) that functions as a transfer device.

[0061] The present invention can also be realized by a process in which one or more processors read and execute a program that realizes one or more functions of the above-described embodiments. The program may be supplied to a system or device having a processor via a network or a storage medium. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions of the above-described embodiments. Each unit of the imaging device 100 may be realized by hardware shown in FIG. 8, or by software.

[0062] Note that one or more functions of the imaging device 100 according to each of the above-described embodiments may be provided in another device. For example, one or more functions of the imaging device 100 according to each of the above-described embodiments may be provided in the imaging device 100 itself. Note that the above-described embodiments may be combined, for example, any combination of the above-described embodiments may be implemented.

[0063] Although the present invention has been described above with reference to the embodiments, the above embodiments merely illustrate specific examples of how the present invention can be implemented, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from the technical concept or main features of the present invention. For example, combinations of the embodiments are also included in the disclosure of this specification. [Explanation of symbols]

[0064] 100 Imaging device 201 Imaging unit 202 Encoding processing unit 203 Detection unit 204 Metadata Generation Unit 205 Distribution Data Generation Unit 206 Transmitter

Claims

1. a detection means for detecting an object included in an image captured by the imaging means; a generating means for generating metadata conforming to ARSEI, the metadata including position information of the object detected from the image by the detecting means; a transmitting means for transmitting to an external device distribution data including encoded data generated by encoding the image and the metadata related to the image, the generating means generates, when a change amount of the parameter of the first object detected from the second image captured after the first image is less than a threshold value, metadata relating to the second image that does not include position information of the first object in the second image; The generation means, when multiple objects are detected from an image, identifies the amount of change in parameters for each object, and selects objects whose location information will be included in the metadata based on the amount of change identified for each object and a set number that is the upper limit of the number of objects to be updated.

2. The imaging device described in claim 1, characterized in that the generation means generates metadata including position information of the first object in the second image as metadata related to the second image when the change in the parameter of the first object is greater than or equal to a threshold.

3. The imaging device described in claim 1 or 2, characterized in that the metadata generated by the generation means for the first image includes ar_object_idx that identifies the first object detected from the first image, and position information and size information of the first object in the first image.

4. 4. The imaging device according to claim 3, wherein the position information corresponds to ar_bounding_box_top and ar_bounding_box_left, and the size information corresponds to ar_bounding_box_width and ar_bounding_box_height.

5. The imaging device described in Claim 4, characterized in that when the change in the parameter of the first object is less than a threshold value, the generation means does not include ar_bounding_box_top, ar_bounding_box_left, ar_bounding_box_width, and ar_bounding_box_height of the first object in the metadata generated for the second image.

6. An imaging device as described in any one of claims 1 to 5, characterized in that the change in the parameter of the first object is an amount based on the size of the area that does not overlap between the area of ​​the first object on the first image and the area of ​​the first object on the second image.

7. An imaging device as described in any one of claims 1 to 5, characterized in that the change in the parameter of the first object is the change between the position of the first object on the first image and the position of the first object on the second image.

8. a detection step of detecting an object included in an image captured by the imaging means; a generating step of generating metadata conforming to ARSEI, the metadata including position information of the object detected from the image in the detecting step; a transmitting step of transmitting to an external device distribution data including encoded data generated by encoding the image and the metadata related to the image, In the generating step, when a change amount of the parameter of the first object detected from the second image captured after the first image is less than a threshold value, metadata related to the second image is generated that does not include position information of the first object in the second image; An information processing method characterized in that, in the generation process, if multiple objects are detected from the image, the amount of change in parameters for each object is identified, and objects whose position information will be included in the metadata are selected based on the amount of change identified for each object and a set number that is the upper limit of the number of objects to be updated.

9. A computer program for causing a computer to function as each of the means included in the imaging device according to any one of claims 1 to 7.

Images