Information processing device, information processing method, and program

Abstract

To reduce the amount of data when expressing a plurality of types of information for a detected object.SOLUTION: An information processing device acquires a plurality of pieces of information about an object detected from an image and ordered in a predetermined order. The information processing device classifies the plurality of pieces of information about the object based on the predetermined order, and controls the information about the object so as to be displayed based on the classification.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device for information relating to an object. [Background technology]

[0002] Technologies are emerging that include information about objects in images as metadata in streams. One example is the ITU-T H.274 Versatile supplemental enhancement information messages for coded video bitstreams standard. This standard includes metadata called ARSEI (Annotated Regions Supplemental Enhancement Information). ARSEI makes it possible to express information about objects by using labels that indicate the type of object recognized by the imaging device and bounding boxes that indicate the object's position.

[0003] When estimating multiple pieces of information about an object from an image and expressing different types of information for a single object, it is necessary to create labels in advance.

[0004] Patent Document 1 discloses a technique in which labels made up of a combination of multiple types are set in advance for one object, and different types of object information are expressed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2020-173562 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the method of setting labels consisting of a plurality of combinations in advance, the amount of data increases as the number of combinations increases.

[0007] The present invention aims to reduce the amount of data when expressing multiple types of information for a detected object. [Means for solving the problem]

[0008] In order to solve the above problems, an information processing device of the present invention has the following configuration: namely, an acquisition means for acquiring a plurality of pieces of information relating to an object detected from an image, the plurality of pieces of information being ordered in a predetermined order, a classification means for classifying the plurality of pieces of information relating to the object based on the predetermined order, and a display control means for controlling the display of information relating to the object based on the classification by the classification means. Effect of the Invention

[0009] According to the present invention, it is possible to reduce the amount of data when expressing a plurality of pieces of information for an object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a configuration of an information processing system according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram showing a part of a data structure of ARSEI according to a first embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram showing an example of object information classification according to an embodiment of the present invention; [Figure 4] 1 is a flowchart showing an information processing method according to a first embodiment of the present invention. [Diagram 5] (a) an image in frame M according to the first embodiment of the present invention, (b) object information in a frame prior to frame M according to the first embodiment of the present invention, and (c) metadata corresponding to frame M according to the first embodiment of the present invention. [Figure 6] 1 is a flowchart showing an information processing method according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0012] <First embodiment> As a first embodiment of the present invention, an example of the configuration of an encoding device according to this embodiment is shown with reference to FIG.

[0013] 1 is a diagram showing the configuration of an image processing system 100 according to this embodiment. The system according to this embodiment includes an encoding device 110, a client device 120, and a network 130 that enables these devices to be interconnected. The network 130 is realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark).

[0014] <Functional configuration of the encoding device 110> Next, an example of the configuration of an encoding device 110 according to this embodiment will be shown with reference to Fig. 1. The encoding device 110 has an image acquisition unit 101, an encoding unit 102, a detection unit 103, a metadata generation unit 104, a communication unit 105, and an internal bus 106 that enables the above components to communicate with each other.

[0015] The image acquisition unit 101 acquires a plurality of images (frames) constituting a moving image from an external imaging unit (not shown). The encoding device 110 may be configured to include an imaging device. The images acquired by the image acquisition unit 101 are not limited to images acquired from an external imaging device (not shown), and may be images stored in an external server or an external memory such as a USB, DVD, or SD card. The image acquisition unit 101 outputs the acquired images to the encoding unit 102 and the detection unit 103.

[0016] The encoding unit 102 encodes the image input from the image acquisition unit 101 using, for example, HEVC, and outputs the encoded image to the communication unit 105. The use of HEVC is just an example, and other encoding methods such as VVC, AV1, and AVC may also be used.

[0017] The detection unit 103 detects moving or stationary objects within a frame in the image input from the image acquisition unit 101, assigns an ID (object ID) that identifies the object, and outputs the ID to the metadata generation unit 104. Note that the image processed by the detection unit 103 may be an image that has been subjected to further image processing such as enlargement, reduction, and format conversion based on the image input from the image acquisition unit 101. Here, the assignment of an object ID means, for example, uniquely associating one of a range of numbers (for example, 1 to 255) with each object detected from consecutive images.

[0018] Furthermore, the detection unit 103 judges the identity between the group of objects detected in the past frame and the object detected in the current frame. For example, an object ID assigned in the past frame may be used as the same object, which will be described later. Here, the judgment of identity may be performed as follows. The detection unit 103 creates an object subgroup, which is a group of objects that are the same as the current object information or have a similar similarity score distribution for multiple object information, from the group of objects detected in the past. The detection unit 103 obtains a velocity vector of the object from the position of each object belonging to the subgroup and the position of the currently detected object, and the detection time of each object belonging to the subgroup and the detection time of the currently detected object. The detection unit 103 judges that the object with the smallest change in the velocity vector between each object assigned an object ID and the currently detected object is the same object.

[0019] Furthermore, the detection unit 103 identifies information about the object (object information). When the detection of the object is completed, the detection unit 103 outputs the object's position information and object information to the metadata generation unit 104. In this embodiment, the detection unit 103 is composed of n detectors, ie, detector 1, detector 2, ..., detector n (n is an integer of 2 or more). Each detector constituting the detection unit 103 detects different types of information as object information. For example, detector 1 detects the color of clothes, such as blue, red, green, and yellow, detector 2 detects the gender, such as male and female, and detector 3 detects the age of the person. The technology for detecting an object in an image and identifying its position and object information is performed using an existing technology, such as template matching. A detection method using AI other than template matching may also be used. In this embodiment, the detection of an object means identifying an object ID, object's position information, and object information.

[0020] 200 in FIG. 2 is a pseudocode of a part of the data structure of ARSEI, and the detection unit 103 outputs the object detection result as ARSEI to the metadata generation unit 104. Here, the fields and syntax related to ARSEI will be briefly explained. The gray columns are the parts related to the control of the data structure, and the white columns are the parts where the actual data is stored. If the data exists, it is decoded according to the gray data structure control. ARSEI has the features that it can update the label ID (201) that specifies the label (202) that can store up to 255 bytes of information, the object ID (203) that identifies the object in the image, and the bounding box that indicates the position of the object. Furthermore, the label corresponding to the object ID can be determined by updating the object ID. The label ID ar_label_idx[i] (201) and the object ID ar_object_idx[i] (203) can register numbers from 0 to 255 (maximum 256) as indexes. Here, the label is ar_label (202), the label ID is ar_label_idx[i] (201), and the object ID is ar_object_idx[i] (203).

[0021] The detection unit 103 judges whether the objects detected in the past frame are the same as the object detected in the current frame, and if it judges that they are the same object, it assigns an object ID to them. For example, the object ID assigned to the object detected in the past frame is also assigned to the object judged to be the same object.

[0022] Furthermore, when the detection unit 103 associates the information detected by each detector with an object ID, the detection unit 103 performs the following process. Specifically, a specific ar_label (202) is associated by inputting the value of ar_label_idx[i] (201) that identifies the ar_label (202) into ar_object_label_idx[i] (205).

[0023] The metadata generating unit 104 generates metadata based on the object position information, object information, and object ID input from the detecting unit 103. In this embodiment, the metadata format is, for example, ARSEI, but other formats may be used. Furthermore, the metadata in this embodiment is basically generated for each frame by the metadata generating unit 105, but if there is no need to update the object information or a label (described later), the metadata does not have to be generated.

[0024] The communication unit 105 generates a stream from the coded data obtained from the coding unit 102 and the metadata obtained from the metadata generation unit 104 , and transmits the stream to the client device 120 via the Internet 130 .

[0025] <Example of functional configuration of client device (information processing device)> The configuration of a client device 120 (information processing device) according to the first embodiment will be described with reference to Fig. 1. The client device 120 has a configuration similar to that of a computer, for example, and includes a control unit 201, a storage unit 202, a label unit 203, a display control unit 204, a communication unit 205, and an internal bus 206 that enables the above configurations to communicate with each other.

[0026] The control unit 201 is composed of, for example, a CPU, and controls the entire client device 120. Furthermore, the control unit 201 decodes images and metadata from a stream acquired via a communication unit 205, and outputs the images to a storage unit 202 and a display control unit 204, and outputs the metadata to a label unit 203.

[0027] The memory unit 202 is used as a storage area for various data such as programs executed mainly by the control unit 201 and information on connectable cameras currently existing on the network 130, and as a work area during program execution. Furthermore, the memory unit 202 associates object information, label ID information identifying the object information, and information on the detector 103 as shown in Fig. 3, and stores them as object classification information. In this embodiment, the object information will be described as ar_label(202) described later, and the ID information identifying the object information will be described as ar_label_idx[i].

[0028] The label unit 203 can identify the object information and the detector information by the object classification information and classify the object information. For example, the label unit 203 can see that 0 to 15 of ar_label_idx[i] are object information related to the color of clothes detected by the detector 1. In this way, it is assumed that the object information detected from the same detector is determined to be the same classification, and that the ID of ar_label_idx[i] is set so that it can be determined. In this embodiment, as shown in FIG. 3, 0 to 15 of ar_label_idx[i] are the first classification, 16 and 17 are the second classification, and 18 to 25 are the third classification, and it can be determined from the order of the ID. In addition, although the object classification information is set in advance in this embodiment, it is not limited to this, and the user may set it later. Furthermore, in this embodiment, the object information, the information of the label ID that identifies the object information, and the information of the detector 103 are associated with each other and stored as object classification information, but it is not limited to this. Information that associates and stores object information and information on a label ID that identifies the object information, and information that associates and stores information on a detector and information on a label ID that identifies the object information may be stored separately.

[0029] The label unit 203 extracts and analyzes the ARSEI from the metadata received from the encoding device 110, and adds and updates object information about the object based on the object classification information obtained from the storage unit 202.

[0030] The process of adding object information to a specific object is performed as follows. If you want to associate a specific ar_label (202) with an object, enter the value of ar_label_idx[i] (201) that identifies the ar_label (202) into ar_object_label_idx[i] (205) and add the object information. The above process is called "adding object information."

[0031] The process of updating object information to new information is performed as follows. By setting ar_object_cancel_flag (204) to 1, the object information of the corresponding object ID can be deleted. The object information can be updated by deleting the object information and adding the object information. The above process is called "updating object information". Note that the determination may be made by executing a specified program without holding the object classification information. For example, the label unit 203 may operate according to a specified program to classify the object information based on the index number, such that 0 to 15 of ar_label_idx[i] are classified as the first classification, 16 and 17 are classified as the second classification, and 18 to 25 are classified as the third classification. In any case, the label unit 203 may classify the object information into each classification based on the index number (ar_label_idx[i]) ordered in a predetermined order, without including information indicating the object classification information separately in the stream.

[0032] A display control unit 205 controls so that the image acquired from the control unit 201 and the object information acquired from the label unit 203 are displayed on a display unit (not shown) including an LED, a liquid crystal display, or the like.

[0033] 1 to 5, the internal configuration of the client device 120 has been described, but the above-mentioned configuration is an example of the configuration of the client device 120 according to the present embodiment, and the present invention is not limited to this. The client device according to the present embodiment may have a configuration including, for example, an image analysis processing unit or an image storage unit, and various modifications and changes are possible within the scope of the gist of the present invention.

[0034] Fig. 4 is a flowchart when the client device 120 according to the first embodiment of the present invention receives metadata and adds object information to an object or updates the object information. The processing in Fig. 4 is performed by the label unit 203 every time the client device 120 is turned on and metadata is periodically received from the encoding device 110. The label unit 204 performs the following processing to add or update object information for the metadata received from the encoding device 110. It is assumed here that this flowchart starts at the timing of processing the metadata associated with frame M.

[0035] In step S401, the label unit 203 acquires ar_object_label_idx[i] from the metadata associated with frame M, and the process proceeds to step S402.

[0036] In step S402, it is determined whether to add or update object information based on the ar_object_label_idx[i] read in step S401 and the object classification information stored in storage unit 202. If the classification of the object information matches, the process proceeds to step S403, and if not, to step S404. Here, the process of determining whether to add or update object information based on the object classification information will be described with reference to Figs. 3 and 5.

[0037] FIG. 5(a) shows an image in frame M, and FIG. 5(b) shows object IDs and object information stored in the storage unit 202. In other words, the object information is made up of metadata acquired before frame M. FIG. 5(c) shows the result of analyzing the metadata newly acquired in step S401. By acquiring ar_object_label_idx[i] (205) in step S401, it is possible to acquire object IDs and object information for identifying objects. Furthermore, this step is progressed by comparing the object IDs and object information stored in the storage unit 202 with the object classification information in FIG. 3. For example, in FIG. 5(c), object information is acquired in which ar_object_label_idx[i] (205) is 15 and 25 for the object with object ID 2. At this time, it can be seen from the object classification information shown in FIG. 3 that i=0 to 15 are detected by detector 1, i=16 to 17 are detected by detector 2, and i=18 to 25 are detected by detector 3. In FIG. 5(b), for object ID 2, object information detected from detector 1 with i=0 and object information detected from detector 2 with i=16 are stored. Furthermore, in step S401, it is assumed that object information detected from detector 1 with i=15 and object information detected from detector 3 with i=25 are acquired. Here, the object information with i=15 acquired in step S401 and the object information with i=0 stored in storage unit 202 are detected from the same detector. In other words, it is determined that the classification of the object information is consistent between ar_object_idx[i] (203) acquired in step S401 and ar_object_idx[i] (205) stored in storage unit 202. In addition, for the object information with i=25 acquired in step S401, object information detected from the same detector is not stored, so it is determined that the classification of the object information is not consistent.

[0038] In step S403, the label unit 204 updates ar_object_label_idx[i] (205) to the ar_object_label_idx[i] (205) acquired in step S401, and proceeds to step S405. That is, the ar_object_label_idx[i] (205) associated with the object ID in a frame prior to frame M is reassociated with the ar_object_label_idx[i] (205) acquired in step S401. That is, in step S403, the object information is updated.

[0039] In step S404, the label unit 204 associates the ar_object_label_idx[i] (205) acquired in step S401 with the ar_object_idx[i] (203). That is, the label unit 204 associates the ar_object_label_idx[i] (205) acquired in step S401 with the ar_object_idx[i] (205), and adds object information.

[0040] In step S405, the object information is output to the storage unit 202 and the display control unit 205, and this flow ends.

[0041] For example, when metadata associated with frame M as shown in FIG. 5(c) is acquired in step S401, the data finally output to storage unit 202 and display control unit 205 in this flow will be described. As shown in FIG. 5(b), according to the object information acquired before frame M, information (red, male) is associated with object ID2. Here, metadata (white, middle-aged) associated with frame M is acquired for object ID2 as shown in FIG. 5(c) in step S401. In step S403, the object information is updated, and object information (white) is associated with object ID2 from object information (red). Furthermore, object information (middle-aged) is newly associated with object ID2 in step S403. Through the above process, object information (white, male, middle-aged) is associated with object ID2 in step S405, and is output to storage unit 202 and display control unit 205.

[0042] In this embodiment, the client device 120 has the above-mentioned configuration, and thus can reduce the amount of data of the object classification information that is set in advance and the amount of data received from the encoding device 110.

[0043] <Second embodiment> The first embodiment showed a method for expressing multiple pieces of information in an object while reducing the amount of data by having the client device 120 update or add object information based on the set object classification information and received metadata.

[0044] In the second embodiment of the present invention, a method of acquiring object classification information and metadata from the encoding device 110 and expressing a plurality of pieces of information in an object while suppressing the amount of data will be described. The information processing system according to the second embodiment of the present invention is a system configured with the encoding device 110 according to the first embodiment of the present invention, a client device 120, and a network 130. Note that similar components are designated by the same numbers, and detailed descriptions thereof will be omitted as they are similar.

[0045] A control method for object information according to this embodiment will be described with reference to the flowchart in Fig. 6. This control flow is processing that starts when the power of the client device 120 is turned on and the client device 120 is connected to the encoding device 110 via the network 130.

[0046] In step S601, the client device 120 obtains object classification information from the encoding device 110. The obtained object classification information is stored in the storage unit 202.

[0047] In step S602, the client device 120 determines whether metadata has been acquired from the encoding device 110. If metadata has been acquired, the process proceeds to step S401, and if metadata has not been acquired, the process of step S602 is repeated.

[0048] In step S603, it is determined whether or not the client device 120 has acquired an end flag. If an end flag has been acquired, this control flow ends, and if not, the process proceeds to step S602.

[0049] In the above, in this embodiment, a method for displaying multiple pieces of object information on an object without setting object classification information in advance by encoding device 110 transmitting object classification information has been described with reference to Fig. 6. The flowchart in Fig. 6 describes an example of internal processing of client device 120 according to this embodiment, and the present invention is not limited to this.

[0050] As described above, by adding or updating object information based on the classification of the object information, it becomes possible to assign a plurality of pieces of object information to one object while suppressing an increase in label resources. [Explanation of symbols]

[0051] 120 Client Device 203 Label section 204 Display control unit 205 Communications Department

Claims

1. An acquisition means for acquiring a plurality of pieces of information related to an object detected from an image, the plurality of pieces of information being ordered in a predetermined order; a classification means for classifying a plurality of pieces of information relating to the object based on the predetermined order; a display control means for displaying information about the object based on the classification by the classification means; having 23. An information processing apparatus comprising:

2. The method further includes: associating means for associating information about an object acquired by the acquiring means with the object; When the associating means associates first information belonging to a first classification with the object, and the acquiring means further acquires second information belonging to the first classification, the associating means associates the second information with the object instead of the first information.

2. The information processing apparatus according to claim 1,

3. When the acquiring means acquires second information belonging to a second classification different from the first classification while the associating means associates first information belonging to the first classification with the object, the associating means associates the first information and the second information with the object.

3. The information processing apparatus according to claim 2.

4. The acquiring means further acquires information regarding an order of the plurality of pieces of information, and the classifying means classifies the plurality of pieces of information regarding the object based on the information regarding the order.

2. The information processing apparatus according to claim 1,

5. the display control means displays information about the object associated with the object by the associating means; 3. The information processing apparatus according to claim 2.

6. 2. The information processing apparatus according to claim 1, wherein the information about the object is included in Annotated Regions Supplemental Enhancement Information.

7. an acquiring step of acquiring a plurality of pieces of information related to the object detected from the image, the plurality of pieces of information being ordered in a predetermined order; a classification step of classifying a plurality of pieces of information related to the object based on the predetermined order; a display control step of controlling display of information about the object based on the classification by the classification step; having 23. An information processing method comprising:

8. A program for causing a computer to execute the information processing method according to claim 7.

Images