Video transmission and receiving system

Abstract

To display appropriate object information even when information on an object that has changed between frames cannot be received.SOLUTION: An video transmission apparatus comprises first communication means for transmitting, when detection means detects that an object has changed between frames, first metadata including information about the changed object together with images of the frames to a video receiving apparatus. The first communication means transmits, for a predetermined frame, second metadata including information about the object in the frame detected by the detection means together with an image of the frame to the video receiving apparatus. The video receiving apparatus includes display control means for, when the first metadata is received, controlling to update information about the object to be displayed based on the first metadata, and, when the second metadata is received, controlling to display information about the object based on the second metadata.SELECTED DRAWING: Figure 19

Description

[Technical Field]

[0001] The present disclosure relates to a video transmission / reception system, a video transmission device, a video reception device, a processing method thereof, and a program. [Background technology]

[0002] The H.265 standard "Annotated Region SEI (ARSEI)" is a technology for adding information about objects in video, i.e., type (label) and bounding box information surrounding the object (rectangle) to a video stream as metadata.

[0003] The labels of objects included in ARSEI-compliant metadata are assigned when they appear, and the bounding boxes retain the same information until their position is updated or they disappear. Labels and bounding boxes are optional in the standard, and active omission is expected to be used to reduce the data volume of the video stream. Patent Document 1 discloses the data structure of ARSEI. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,951,903 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in the above-mentioned Patent Document 1 mainly discloses the data structure of metadata, and does not describe in detail how to operate it.

[0006] For example, if the sender and receiver of a video stream are connected via a network via the Internet, it is necessary to assume that the network communication bandwidth will be constrained by communications other than those of the video stream sender and receiver. If the video stream uses a UDP-based protocol such as RDP, it must be acceptable for the receiver to lose video stream data due to network congestion.

[0007] However, data loss, i.e., the lack of ARSEI-compliant metadata, can lead to abnormal object display. Specifically, if metadata is lost when an object appears, the object will have an unknown label. If metadata is lost when an object disappears, the object's label and bounding box (the display frame surrounding the object) will remain.

[0008] An object of the present disclosure is to make it possible to display information about an appropriate object even when information about an object that has changed between frames cannot be received. [Means for solving the problem]

[0009] The video transmission / reception system includes a video transmission device and a video receiving device. The video transmission device has a detection means for detecting an object included in each frame of a plurality of frame images, and a first communication means for, when the detection means detects that an object has changed between frames, transmitting first metadata including information about the changed object to the video receiving device together with the frame image. The first communication means transmits second metadata including information about the object in a given frame detected by the detection means to the video receiving device together with the frame image. The video receiving device has second communication means for receiving the frame image together with the first metadata in one frame and receiving the frame image together with the second metadata in another frame, and a display control means for, when the first metadata is received, controlling to update information about the object to be displayed based on the first metadata, and for, when the second metadata is received, controlling to display information about the object based on the second metadata. [Effects of the Invention]

[0010] According to the present disclosure, even if information about an object that has changed between frames cannot be received, appropriate object information can be displayed. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a video transmission / reception system. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of a video transmitting device and a video receiving device. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of metadata. [Figure 4] 10 is a flowchart showing metadata creation. [Figure 5] 1 is a flowchart showing metadata analysis. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of metadata. [Figure 7]FIG. 10 is a diagram showing an example of an area configuration within a screen of extended metadata. [Figure 8] 10 is a flowchart showing extended metadata creation. [Figure 9] 10 is a flowchart showing the creation of area-specific metadata. [Figure 10] 1 is a flowchart illustrating extended metadata analysis. [Figure 11] FIG. 2 is a diagram illustrating an example of the configuration of metadata. [Figure 12] 10 is a flowchart showing metadata creation. [Figure 13] 10 is a flowchart showing standard metadata creation. [Figure 14] 10 is a flowchart showing extended metadata creation. [Figure 15] 10 is a flowchart illustrating extended metadata labeling. [Figure 16] 1 is a flowchart illustrating extended metadata analysis. [Figure 17] FIG. 10 is a diagram illustrating an example of a video stream. [Figure 18] FIG. 10 is a diagram illustrating an example of metadata. [Figure 19] FIG. 10 is a diagram illustrating an example of a video stream. DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments will be described in detail below with reference to the drawings.

[0013] (First embodiment) 1 is a diagram showing an example of the configuration of a video transmission and reception system 1500 according to the first embodiment. The video transmission and reception system 1500 includes a video transmission device 1000, a network 2000, and a video reception device 3000.

[0014] The video transmitting device 1000 is connected to the video receiving device 3000 via a network 2000. The video receiving device 3000 receives a video stream conforming to the H.265 standard from the video transmitting device 1000.

[0015] The video transmission device 1000 according to the first embodiment will be described below with reference to Fig. 2(a). Fig. 2(a) is a diagram showing an example of the configuration of the video transmission device 1000 of Fig. 1.

[0016] The video transmission device 1000 includes an imaging unit 1010, a communication unit 1020, a control unit 1030, a storage unit 1040, a compression encoding unit 1050, and an object detection unit 1060. The imaging unit 1010 includes an optical system (lens) 1011, an image sensor 1012, and a development processing unit 1013.

[0017] 2(a), light rays from a subject to be photographed pass through an optical system 1011 and form an image on an image sensor 1012. The image sensor 1012 captures an image of the subject and outputs it to a development processing unit 1013 as a video signal.

[0018] The communication unit 1020 is connected to a network 2000 such as Ethernet, and receives image size and frame rate setting instructions from the video receiving device 3000, as well as image-related setting commands such as exposure control and white balance for the subject image.

[0019] The image setting command is analyzed by the control unit 1030 and input to the image sensor 1012 and the development processing unit 1013. At the same time, image setting information is stored in the storage unit 1040, and the image sensor 1012 and the development processing unit 1013 are set by the control unit 1030 in accordance with the image setting information stored in the storage unit 1040 at startup.

[0020] The development processing unit 1013 performs development processing on the video signal from the image sensor 1012 in accordance with the image setting information, and outputs the resulting image to the compression encoding unit 1050 and the object detection unit 1060.

[0021] 2(a), the communication unit 1020 receives setting commands such as the compression code amount of the transmission video stream and transmission start / stop commands from the video receiving device 3000. The setting commands for the transmission video stream are analyzed by the control unit 1030 and input to the compression encoding unit 1050. At the same time, transmission video stream setting information is stored in the storage unit 1040, and the control unit 1030 sets the compression encoding unit 1050 in accordance with the transmission video stream setting information stored in the storage unit 1040 at the time of startup.

[0022] The transmission start command is analyzed by the control unit 1030, and the communication unit 1020 starts transmitting the video stream compressed by the compression encoding unit 1050 in H.265 to the video reception device 3000 that has requested the transmission.

[0023] The transmission stop command is analyzed by the control unit 1030, and the communication unit 1020 stops transmitting the video stream compressed by the compression encoding unit 1050 in H.265 to the video receiving device 3000.

[0024] 2(a), the communication unit 1020 receives an object detection setting command from the video receiving device 3000. The object detection setting is setting information for limiting the area of ​​object detection information to be received among objects detected from the developed image, and for switching the dictionary and neural network of objects to be detected.

[0025] The object detection setting command is analyzed by control unit 1030 and input to object detection unit 1060. At the same time, object detection setting information is stored in memory unit 1040. At startup, control unit 1030 sets object detection unit 1060 in accordance with the object detection setting information stored in memory unit 1040.

[0026] The object detection unit 1060 detects objects in the image input from the development processing unit 1013. The object detection information detected by the object detection unit 1060 is read by the control unit 1030, processed into metadata of an H.265 compressed video stream, and transmitted from the communication unit 1020 to the video receiving device 3000 together with the compressed image.

[0027] Fig. 2(b) is a diagram showing an example of the configuration of the video receiving device 3000 of Fig. 1. The video receiving device 3000 has a communication unit 1110, a display unit 1120, a control unit 1130, a storage unit 1140, and an input unit 1150.

[0028] The communication unit 1110 connects to the network 2000 and transmits image setting commands such as instructions for setting the image size and frame rate, and setting commands related to images such as exposure control and white balance for the subject image, to the video transmission device 1000. The image setting commands are generated by the control unit 1030 in response to user operations on the input unit 1150.

[0029] The communication unit 1110 transmits setting commands such as the compression code amount of the video stream to be transmitted and transmission start / stop commands to the video transmitting device 1000. The setting commands for the video stream to be transmitted are generated by the control unit 1130.

[0030] The transmission start command is generated by the control unit 1130. The communication unit 1110 can start receiving the compressed video stream from the video transmitting device 1000.

[0031] The transmission stop command is generated by the control unit 1130. The communication unit 1110 stops receiving the compressed video stream from the video transmitting device 1000.

[0032] The communication unit 1110 transmits a setting command for object detection to the video transmission device 1000. The setting command for object detection is generated by the control unit 1130.

[0033] The communication unit 1110 receives the metadata together with the compressed image from the video transmitting device 1000 .

[0034] The storage unit 1140 stores metadata, compressed images, programs, etc. The control unit 1130 is, for example, a CPU, and executes the programs stored in the storage unit 1140.

[0035] The control unit 1130 decodes the compressed image and displays it on the display unit 1120, and also superimposes and displays an object on the display unit 1120 based on the object detection information in the metadata.

[0036] Here, with reference to FIG. 3, a data structure 4000 of metadata in this embodiment will be described.

[0037] The control unit 1030 in this embodiment generates metadata of a data structure 4000 conforming to ARSEI as shown in FIG.

[0038] ARSEI is the H.265 standard "Annotated Region SEI (ARSEI)."

[0039] 3 is a data structure of ARSEI written as pseudocode. In the data structure 4000, the hatched columns indicate the parts related to the control of the data structure.

[0040] In the data structure 4000, the columns without hatching are the portions where actual data is stored, if data exists, according to the hatched data structure control.

[0041] A feature of ARSEI is that it can update ar_label_idx[i] (4002), which identifies ar_label (4001), which can store up to 255 bytes of information (such as text).

[0042] It is also possible to update the ar_object_idx[i] (4003), which is an object index for identifying an object in an image, and information indicating the position of the object.

[0043] Up to 256 ar_label_idx[i] (4002) and ar_object_idx[i] (4003) can be registered, and it is possible to determine which label to assign to an object as the object is updated.

[0044] Here, when the ar_label (4001) is updated based on the object detection information detected by the object detection unit 1060, the number of updated labels, which is 1 or more, is stored in the ar_num_label_updates (4004).

[0045] The ar_label (4001) can be updated by storing the index (any value between 0 and 255) associated with the ar_label (4001) in ar_label_idx (4002). The ar_label (4001) can store non-zero data up to 255 bytes.

[0046] When updating an object based on object detection information detected by the object detection unit 1060, the number of objects to be updated, which is equal to or greater than 1, is stored in ar_num_object_updates (4005). Then, an index (any of 1 to 255) associated with the object to be updated is stored in ar_object_idx (4003).

[0047] If you want to associate a specific ar_label (4001) with the object you want to update, set ar_object_label_update_flag (4006) to 1. Then, enter the value of ar_label_idx (4002) that identifies ar_label (4001) in ar_object_label_idx (4007).

[0048] Furthermore, by setting ar_bounding_box_update_flag (4008) to 1, it is possible to store bounding box information 4009 that indicates the position and size of the object to be updated. ar_object_confidence (4010) can also be updated.

[0049] Note that bounding box information 4009 indicating the position and size of the object is described by ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]], as shown in FIG. 3.

[0050] The ar_bounding_box_top[ar_object_idx[i]] and ar_bounding_box_left[ar_object_idx[i]] syntax elements indicate the position of the bounding box of an object on the image.

[0051] Furthermore, the ar_bounding_box_width[ar_object_idx[i]] syntax element indicates the width of the bounding box of an object on the image, and the ar_bounding_box_height[ar_object_idx[i]] syntax element indicates the height of the bounding box of the object on the image.

[0052] By setting the ar_label_cancel_flag (4011) for a certain ar_label_idx (4002) to 1, the information of the ar_label (4001) linked to the ar_label_idx (4002) can be deleted.

[0053] Furthermore, by setting the ar_object_cancel_flag (4012) for a certain numerical value of ar_object_idx (4003) to 1, information about the object of the ar_object_idx (4003) can be deleted.

[0054] The information about the object includes bounding box information 4009 and ar_object_confidence (4010).

[0055] Furthermore, by setting the ar_bounding_box_cancel_flag (4012) for a certain ar_object_idx (4003) to 1, the bounding box information 4009 etc. corresponding to the ar_object_idx (4003) can be deleted.

[0056] The data structure 4000 conforming to ARSEI has been described above. Below, a data structure in this embodiment that extends ARSEI will be described.

[0057] Statistical information about objects in an image based on object detection information detected by the object detection unit 1060 can be stored as ar_object_summary (4013) by entering index 0 into ar_object_idx (4003).

[0058] ar_object_summary(4013) is 32 bytes, i.e., 255 bits of data, and there is a one-to-one correspondence between the bit position and ar_object_idx(4003). In other words, when bit X of ar_object_summary(4013) is 1, an object corresponding to ar_object_idx(4003)==X exists in the image. When bit X of ar_object_summary(4013) is 0, it means that an object corresponding to ar_object_idx(4003)==X does not exist.

[0059] A normal object is described with an index of 1 or more in ar_object_idx (4003).

[0060] A procedure for creating metadata including statistical information of objects in an image on the side of the video transmitting device 1000 in this embodiment will be described below with reference to Fig. 4. A processing method of the video transmitting device 1000 will be described below.

[0061] 4 is a flowchart showing a metadata creation method of the video transmitting device 1000. The control unit 1030 is, for example, a CPU. The control unit 1030 executes a program stored in the storage unit 1040 to realize the processing of the flowchart in FIG.

[0062] In step S101, the control unit 1030 stores index 0 at the beginning of the array of ar_object_idx (4003). That is, the control unit 1030 stores 0 in ar_object_idx[0].

[0063] In step S102 following step S101, control unit 1030 determines whether an object has been detected for index X (1 to 255) associated with the object in the object detection information detected by object detection unit 1060. If an object specified by index X has not been detected in step S102, the process proceeds to step S103. If an object specified by index X has been detected, the process proceeds to step S104.

[0064] In step S103, the control unit 1030 clears to 0 the bit X corresponding to the index X in the statistical information ar_object_summary (4013) of the object in the image.

[0065] In step S104, the control unit 1030 sets bit X corresponding to index X in ar_object_summary (4013) to 1.

[0066] In step S105 following step S104, the control unit 1030 adds index X to the i-th position of the array of ar_object_idx (4003), where i is a value equal to or less than ar_num_object_updates (4005), which monotonically increases each time the process proceeds to step S104 while an object is present on the screen.

[0067] In step S106, the control unit 1030 sets ar_object_cancel_flag (4012) to 0 and ar_object_label_update_flag (4006) to 1.

[0068] This makes it possible to store the label of the object specified by index X in the metadata.

[0069] Step S107 is the last step in processing the object specified by index X. In step S107, the control unit 1030 updates the label number of the object corresponding to index X that identifies ar_label (4001) at the i-th position in the array of ar_object_label_idx (4007).

[0070] The procedure for analyzing metadata including statistical information of objects in an image on the video receiving device 3000 side in this embodiment will be described below with reference to Fig. 5. The processing method of the video receiving device 3000 will be described below.

[0071] 5 is a flowchart showing a metadata analysis method of the video receiving device 3000. The control unit 1130 is, for example, a CPU. The control unit 1130 executes a program stored in the storage unit 1140 to realize the processing of the flowchart in FIG.

[0072] In step S201, the control unit 1130 determines whether the i-th index of the array of ar_object_idx (4003) is 0. If the i-th index is not 0, the process proceeds to step S202 of the object update process of the ARSEI standard. If the i-th index is 0, the process proceeds to step S203 of the determination process of ar_object_summary (4013).

[0073] In step S202, the control unit 1130 updates the object indicated by ar_object_idx[i] (4003) in accordance with the data of ARSEI, and ends the i-th process of ar_object_idx (4003).

[0074] In step S203, the control unit 1130 determines whether bit X corresponding to index X of ar_object_summary (4013) is 0. If bit X is 1, the control unit 1130 immediately ends processing for index X of ar_object_summary (4013). If bit X is 0, the control unit 1130 proceeds to step S204.

[0075] In step S204, the control unit 1130 deletes the bounding box information 4009 of object X, etc., since it indicates that object X associated with ar_object_idx[i](4003)==X does not exist within the screen.

[0076] As described above, even if metadata including ar_label_cancel_flag (4011) is lost on the communication path, it is possible to delete the bounding box information 4009 of the object by using ar_object_summary (4013).

[0077] In this embodiment, the ar_object_label_idx (4007) is resent together with the ar_object_summary (4013). This means that even if the ar_object_label_idx (4007) is missing and an incorrect ar_label (4001) is assigned, it can be corrected together with the ar_label_cancel_flag (4011).

[0078] Similar to ar_object_label_idx (4007), it is possible to resend the bounding box information 4009 together with the ar_object_summary (4013).

[0079] The timing for inserting the ar_object_summary (4013) may be the timing of an I frame of H.265 or the timing of a P frame before or after the I frame. The ar_object_summary (4013) may be inserted by receiving an insertion command for the ar_object_summary (4013) from the video reception device 3000.

[0080] As described above, this embodiment is characterized in that in step S101 of FIG. 4, the control unit 1030 adds 0 to the beginning of the array of ar_object_idx (4003) to store it as ar_object_summary (4013).

[0081] To explain the effect of this embodiment, first, a case where there is no ar_object_summary (4013) will be explained.

[0082] FIG. 17(a) is a diagram showing an example of a video stream when there is no ar_object_summary (4013).

[0083] The control unit 1030 of the video transmitting device 1000 transmits frames at a predetermined frame rate. An I frame is a frame compressed using the information within the frame. A P frame is a frame compressed by compressing only the difference from the previous frame.

[0084] The control unit 1030 also transmits metadata SEI1 to SEI4 along with the frames. For example, the control unit 1030 transmits an I frame and metadata SEI1 in the first frame, and transmits a P frame and metadata SEI2 in the fourth frame. The control unit 1030 also transmits a P frame and metadata SEI3 in the sixth frame, and transmits a P frame and metadata SEI4 in the eighth frame.

[0085] Metadata SEI1 indicates that the object detection unit 1060 has detected the appearance of object 1 and object 2.

[0086] Metadata SEI2 indicates that the object detection unit 1060 has detected a rectangle update for object 2.

[0087] Metadata SEI3 indicates that the object detection unit 1060 has detected the disappearance of object 2.

[0088] Metadata SEI4 indicates that the object detection unit 1060 has detected the disappearance of object 1.

[0089] The control unit 1130 of the video receiving device 3000 validates the appearance information of object 1 in the metadata SEI1 for object 1 in the frames up until the metadata SEI3 is received.

[0090] Furthermore, the control unit 1130 of the video receiving device 3000 enables the appearance information of object 2 in the metadata SEI1 for object 2 in the frames up until the metadata SEI2 is received.

[0091] In this way, ARSEI allows object information (type=label, rectangle information=boundary box) within a screen to be included in a video stream as metadata SEI1 to SEI4.

[0092] An object label is assigned when the object appears. The bounding box continues to have the same information until the position is updated or it disappears. To reduce the amount of data, it is not included in frames that do not change.

[0093] 17(b) is a diagram showing an example of a video stream when the video receiving device 3000 is unable to receive frame 6. There are cases where the video receiving device 3000 is unable to receive frames of the video stream due to congestion on the network 2000.

[0094] For example, the video receiving device 3000 cannot receive the sixth frame and the metadata SEI3. As a result, the video receiving device 3000 determines that the appearance information of object 1 in the metadata SEI1 is valid in the first to ninth frames, and is unable to recognize the disappearance of object 1.

[0095] In this way, if a frame is dropped during video communication and object information is lost, an abnormal object may appear in the video receiving device 3000. Specifically, if a frame in which an object appears is dropped, the object becomes one with an unknown label. Furthermore, if a frame in which an object is lost is dropped, a phenomenon occurs in which the object's rectangular information (a frame display indicating its position) continues to remain. Therefore, the present embodiment aims to detect and correct problematic objects.

[0096] FIG. 18 shows examples of metadata 1801 when an object appears, metadata 1802 when an object moves, and metadata 1803 when an object disappears.

[0097] Metadata 1801 is metadata when the object in Fig. 17(a) appears. The metadata 1801 includes ar_object_idx[i] (4003), ar_object_cancel_flag (4014), ar_object_label_idx (4007), and bounding box information 4009 in Fig. 3. Based on the metadata 1801, the video receiving device 3000 displays information such as the bounding box (a frame indicating the position) and label of the object at a predetermined position.

[0098] Metadata 1802 is metadata when the object moves (position is updated) in Fig. 17(a). The metadata 1802 includes ar_object_idx[i] (4003), ar_object_cancel_flag (4014), and bounding box information 4009 in Fig. 3. The video receiving device 3000 updates the display position of the object information based on the metadata 1802.

[0099] Metadata 1803 is metadata when the object in Fig. 17(a) disappears. The metadata 1803 includes ar_object_idx[i] (4003) and ar_object_cancel_flag (4014) in Fig. 3. The video receiving device 3000 hides the object information based on the metadata 1803.

[0100] The video transmitting device 1000 does not transmit all information every time an object is detected, but rather does not transmit information that has not changed.

[0101] FIG. 19 is a diagram showing an example of a video stream when the ar_object_summary (4013) in FIG. 4 is present.

[0102] Fig. 19 is obtained by adding metadata SEI5 to Fig. 17(a). Metadata SEI1 to SEI4 indicate normal objects corresponding to metadata 1801 to 1803 in Fig. 19, and an index of 1 or more is described in ar_object_idx (4003).

[0103] The metadata SEI5 indicates ar_object_summary (4013) corresponding to step S101 in FIG. 4, and an index of 0 is written in ar_object_idx (4003).

[0104] For example, in the tenth frame, metadata SEI5 includes information on the disappearance of object 1 and object 2 in FIG.

[0105] As shown in step S103 of FIG. 4, if object X does not exist, bit X corresponding to index X in ar_object_summary (4013) becomes 0.

[0106] As shown in step S104 of FIG. 5, if object X exists, bit X corresponding to index X in ar_object_summary (4013) becomes 1.

[0107] As a result, even if the video receiving device 3000 is unable to receive the metadata SEI3, by receiving the metadata SEI5, it can recognize the disappearance of object 1 and hide the information about object 1 (S204 in Figure 5).

[0108] The video transmitting device 1000 can transmit the metadata SEI5, for example, at the timing of an I frame, at the timing of a P frame immediately before an I frame, or at the timing of a P frame immediately after an I frame.

[0109] As described above, the object detection unit 1060 in FIG. 2(a) detects an object included in each frame of a plurality of frame images.

[0110] When the object detection unit 1060 detects that an object has changed between frames, the control unit 1030 transmits, via the communication unit 1020, first metadata SEI1 to SEI4 containing information about the changed object to the video receiving device 3000 together with the image of the frame.

[0111] 4, the control unit 1030 generates, for a given frame, second metadata SEI5 including information about objects in the frame detected by the object detection unit 1060. The second metadata SEI5 includes information ar_object_summary (4013) indicating the presence or absence of an object in the frame for each object index, as in steps S103 and S104 of FIG. 4. Then, the control unit 1030 transmits the second metadata SEI5, together with an image of the frame, to the video receiving device 3000 via the communication unit 1020.

[0112] The control unit 1130 in FIG. 2(b) receives, via the communication unit 1110, an image of a frame together with first metadata SEI1 to SEI4 for a certain frame, and receives, via the communication unit 1110, an image of the frame together with second metadata SEI5 for another frame.

[0113] In step S201 of FIG. 5, if the control unit 1130 receives the first metadata SEI1 to SEI4, the process proceeds to step S202, and if the control unit 1130 receives the second metadata SEI5, the process proceeds to step S203.

[0114] In step S202, the control unit 1130 functions as a display control unit, and performs control so as to update information about objects to be displayed based on the first metadata SEI1 to SEI4, as shown in Fig. 18. Specifically, the control unit 1130 performs control so as to update information about objects to be superimposed on the image of the received frame based on the first metadata SEI1 to SEI4.

[0115] From step S203 onwards, the control unit 1130 performs control to display information about the object based on the second metadata SEI 5. Specifically, the control unit 1130 performs control to superimpose information about the object on the received frame based on the second metadata SEI 5.

[0116] For example, a case where the sixth frame cannot be received as shown in Fig. 17(b) will be described as an example. As shown in Fig. 19, the control unit 1030 transmits first metadata SEI1 indicating the appearance of a first object, then transmits first metadata SEI3 indicating the disappearance of the first object, and then transmits second metadata SEI5 indicating the absence of the first object.

[0117] If the control unit 1130 receives first metadata SEI1 indicating the appearance of a first object, but does not receive first metadata SEI3 indicating the disappearance of the first object, and instead receives second metadata SEI5 indicating the absence of the first object, then in step S204 of FIG. 5, the control unit 1130 performs control so that information about the first object is not displayed.

[0118] Next, the timing of transmitting the second metadata SEI5 will be described. The control unit 1030 transmits the second metadata SEI5 together with the image of the I frame, the image of the P frame immediately before the I frame, or the image of the P frame immediately after the I frame.

[0119] Next, step S201 in Fig. 5 will be described. In the first metadata SEI1 to SEI4 in Fig. 19, the ar_object_idx (4003) of each object is identified by an index of a value equal to or greater than 1. In the second metadata SEI5, the ar_object_idx (4003) of the object has an index of 0.

[0120] In step S201, if the control unit 1030 receives metadata in which the object index is a value of 1 or greater, it recognizes that the received metadata is first metadata SEI1 to SEI4, and proceeds to step S202. On the other hand, if the control unit 1030 receives metadata in which the object index is 0, it recognizes that the received metadata is second metadata SEI5, and proceeds to step S203.

[0121] According to this embodiment, even if an error occurs in receiving the first metadata SEI3 indicating the appearance or disappearance of an object, by transmitting the second metadata SEI5, it is possible to prevent the display of information about an incorrect object and display information about the appropriate object.

[0122] (Second embodiment) Hereinafter, the data structure 5000 of the metadata of the video transmitting device 1000 according to the second embodiment will be described with reference to FIG.

[0123] Fig. 6 is a diagram showing an example of a metadata data structure 5000. The data structure 5000 shown in Fig. 6 is the data structure of ARSEI written as pseudocode.

[0124] In data structure 5000, hatched columns indicate the portions related to the control of the data structure, and non-hatched columns indicate the portions where actual data is stored, if data exists, according to the hatched data structure control.

[0125] The following describes a data structure that extends ARSEI in this embodiment. Statistical information for each of the four areas of an object in an image, based on object detection information detected by the object detection unit 1060, is specified by entering index 0 into the array ar_object_idx (5001). Statistical information for each of the four areas is stored as ar_object_summary_areaN (N=0, 1, 2, 3) (5002).

[0126] The ar_object_summary_areaN (N=0, 1, 2, 3) (5002) is 32 bytes, i.e., 255 bits of data, and the bit position corresponds one-to-one to the ar_object_idx (5001).

[0127] That is, when bit X of ar_object_summary_areaN (N=0, 1, 2, 3) (5002) is 1, an object corresponding to ar_object_idx (5001)==X exists in the image.

[0128] When bit X of ar_object_summary_areaN (N=0, 1, 2, 3) (5002) is 0, it indicates that there is no object corresponding to ar_object_idx (5001)==X.

[0129] A normal object is described with an index of 1 or more in ar_object_idx (5001).

[0130] In this embodiment, the configuration of the four areas is as shown in Fig. 7. ar_object_summary_area_info (5003) is information on the correspondence between a to d in the divided area 6000 in Fig. 7 and ar_object_summary_areaN (N = 0, 1, 2, 3) (5002).

[0131] There are 24 possible combinations of ar_object_summary_areaN (N=0, 1, 2, 3) (5002), and ar_object_summary_area_info (5003) is expressed in 1 byte.

[0132] Hereinafter, with reference to FIGS. 8 and 9, a procedure for creating statistical information ar_object_summary_areaN (N=0, 1, 2, 3) (5002) of an object for each image area in the video transmission device 1000 according to this embodiment will be described.

[0133] 8 is a flowchart showing a method for creating extended metadata by the video transmitting device 1000. The control unit 1030 executes a program stored in the storage unit 1040 to realize the processing of the flowchart in FIG.

[0134] In step S301, the control unit 1030 stores index 0 in the i-th position of the array of ar_object_idx (5001), where i is a value equal to or less than ar_num_object_updates (5004) described in the metadata.

[0135] In step S302 following step S301, the control unit 1030 creates statistical information ar_object_summary_areaN (N=0, 1, 2, 3) (5002) for each of the four areas of the object in the image, and stores it as metadata.

[0136] In step S303 following step S302, the control unit 1030 stores the configuration information ar_object_summary_area_info (5003) of the four areas in the image as metadata.

[0137] 9 is a flowchart showing a method for creating area-specific metadata by the video transmitting device 1000. The control unit 1030 executes a program stored in the storage unit 1040 to realize the processing of the flowchart in FIG.

[0138] In step S401, the control unit 1030 determines whether or not an object has been detected in the area N for the index X (1 to 255, respectively) associated with the object in the object detection information detected by the object detection unit 1060.

[0139] Whether or not an object exists in area N is determined based on whether or not the center of the object indicated by the bounding box information of the object detected by object detection unit 1060 is within area N. Alternatively, if the rectangle based on the bounding box information of the object detected by object detection unit 1060 overlaps with area N even partially, it may be determined that the object exists in area N.

[0140] If the object specified by index X is not detected in step S401, the process proceeds to step S402. If the object specified by index X is detected, the process proceeds to step S403.

[0141] In step S402, the control unit 1030 clears to 0 the bit X corresponding to the index X of the statistical information ar_object_summary_areaN (5002) of the object in the image with index 0.

[0142] In step S403, the control unit 1030 sets bit X corresponding to index X of ar_object_summary_areaN (5002) to 1.

[0143] Hereinafter, with reference to FIG. 10, a procedure for analyzing statistical information ar_object_summary_areaN (N=0, 1, 2, 3) (5002) of an object for each image area in the video receiving device 3000 according to this embodiment will be described.

[0144] 10 is a flowchart showing an extended metadata analysis method of the video receiving device 3000. The control unit 1130 executes a program stored in the storage unit 1140 to realize the processing of the flowchart in FIG.

[0145] In step S501, the control unit 1130 determines whether or not there is an object X corresponding to ar_object_idx[i](5001)==X in the object information held by the video receiving device 3000. If there is no object X in step S501, the process proceeds to step S502. If there is an object X in step S501, the process proceeds to step S503.

[0146] In step S502, the control unit 1130 determines whether bit X corresponding to ar_object_idx (5001) == X in ar_object_summary_areaN (N=0, 1, 2, 3) (5002) is zero or not. If bit X is zero in step S502, the process in Fig. 10 ends. If bit X is not zero in step S502, the process proceeds to step S504.

[0147] In step S504, the control unit 1130 changes the label information of object X held by the video receiving device 3000 to unknown, and the process proceeds to step S508.

[0148] In step S503, the control unit 1130 determines whether bit X corresponding to ar_object_idx[i] (5001) == X in ar_object_summary_areaN (N=0,1,2,3) (5002) is zero or not. If bit X is not zero in step S503, the process proceeds to step S505. If bit X is zero in step S503, the process proceeds to step S506.

[0149] In step S505, the control unit 1130 determines whether or not the object X corresponding to ar_object_idx[i] (5001)==X is in the area N indicated by ar_object_summary_areaN (N=0, 1, 2, 3) (5002).

[0150] Whether or not object X is in area N is determined by the method described in the determination method of step S401 in Fig. 9. If object X is in area N, the processing of Fig. 10 ends. If object X is not in area N, the process proceeds to step S507.

[0151] In step S507 following step S505, the control unit 1130 changes the bounding box information of object X held by the video receiving device 3000 to area N, and the process proceeds to step S508.

[0152] In step S506, the control unit 1130 determines that the ar_label_cancel_flag of object X associated with ar_object_idx(5001)==X has been lost on the communication path, and deletes the object information of object X.

[0153] In step S508, which follows step S504 or S507, the control unit 1130 sets warning information for object X held by the video receiving device 3000. When drawing object X in an image on the display screen of the display unit 1120, the warning information is used to draw missing metadata on the display screen by highlighting the label of object X or by changing the bounding box to a warning color.

[0154] The above shows that after the object information of object X linked to ar_object_idx[i] (5001) == X is lost on the communication path, the object information can be corrected using ar_object_summary_areaN (N = 0, 1, 2, 3) (5002).

[0155] The timing of inserting ar_object_summary_areaN (N=0, 1, 2, 3) (5002) is the same as in the first embodiment.

[0156] As in Figures 7 to 9, the second metadata SEI5 in Figure 19 includes, for each of areas a to d in the frame, information ar_object_summary_areaN (5002) indicating the presence or absence of an object in the frame for each object index in steps S402 and S403 in Figure 9.

[0157] Next, an example of a reception error that occurs when an object has moved will be described. After transmitting first metadata SEI1 indicating that the position of a first object is a first region a within a frame, the control unit 1030 transmits first metadata SEI2 indicating that the first object has moved, and then transmits second metadata SEI5 indicating that the first object is not in the first region a but is present in a second region b different from the first region a.

[0158] If the control unit 1130 receives first metadata SEI1 indicating that the position of the first object is in a first region a within the frame, but does not receive first metadata SEI2 indicating the movement of the first object, and instead receives second metadata SEI5 indicating that the first object is not in the first region a but is present in a second region b different from the first region a, the control unit 1130 performs control to display the information about the first object in the second region as in step S507 of FIG. 10, rather than displaying the information about the first object in the first region a as in step S506 of FIG. 10.

[0159] In this case, the control unit 1130 performs control so that a warning is displayed in the area of ​​the first object, as in step S508 in FIG.

[0160] (Third embodiment) Hereinafter, with reference to FIG. 11, a data structure 7000 of metadata of the video transmitting device 1000 according to the third embodiment will be described.

[0161] 11 shows a data structure 7000 in which the data structure of ARSEI is written as pseudocode. In the data structure 7000, the hatched columns indicate the parts related to the control of the data structure.

[0162] In the data structure 7000, the columns without hatching are areas where actual data is stored, if data exists, according to the hatched data structure control.

[0163] A data structure 7000 that extends ARSEI in this embodiment will be described below. Statistical information on objects present in an image based on object detection information detected by the object detection unit 1060 can be stored as ar_object_summary (7002) by entering index 0 into ar_object_idx (7001).

[0164] Also, statistical information on objects that have disappeared in chronological order between the previous ar_object_summary (7002) and the current ar_object_summary (7002) is stored as ar_object_summary_canceled (7003).

[0165] A normal object is described with an index of 1 or more in ar_object_idx (7001).

[0166] The ar_object_summary (7002) is 32 bytes, that is, 255 bits of data, and the bit positions correspond one-to-one to the ar_object_idx (7001).

[0167] In other words, when bit X of ar_object_summary (7002) is 1, an object corresponding to ar_object_idx (7001) == X exists in the image. When bit X of ar_object_summary (7002) is 0, an object corresponding to ar_object_idx (7001) == X does not exist.

[0168] The ar_object_summary_canceled (7003) is 32 bytes, that is, 255 bits of data, and the bit positions correspond one-to-one to the ar_object_idx (7001).

[0169] In other words, when bit X of ar_object_summary_canceled (7003) is 1, it is a history of object X corresponding to ar_object_idx (7001)==X disappearing from the image.

[0170] When bit X of ar_object_summary_canceled (7003) is 0, it is a history that object X corresponding to ar_object_idx (7001) == X has not disappeared from the image.

[0171] The ar_object_summary_label_idx[ar_object_idx[i]] (7004) is an index that identifies the ar_label (7005) of the object X corresponding to ar_object_idx[i] == X. The number of elements in the ar_object_summary_label_idx (7004) is the number of 1 bits in the logical product of the ar_object_summary (7002) and ar_object_summary_canceled (7003).

[0172] Hereinafter, with reference to Figs. 12 to 15, a procedure for creating the statistical information ar_object_summary (7002) and ar_object_summary_canceled (7003) of objects in an image by the video transmission device 1000 in this embodiment will be described.

[0173] 12 is a flowchart showing a metadata creation method of the video transmitting device 1000. The control unit 1030 executes a program stored in the storage unit 1040 to realize the processing of the flowchart in FIG.

[0174] In step S601, the control unit 1030 determines whether to insert ar_object_summary (7002) and ar_object_summary_canceled (7003). The timing for inserting ar_object_summary (7002) and ar_object_summary_canceled (7003) is the timing of an H.265 I frame, the timing of the P frame immediately before the I frame, or the timing of the P frame immediately after the I frame. It may also be the timing when an ar_object_summary (7002) insertion command is received from the video receiving device 3000. Furthermore, since the amount of metadata data increases when an object appears, moves, or disappears within an image, insertion may be suppressed to reduce the amount of communication data. If not, proceed to step S602; if inserted, proceed to step S603.

[0175] In step S602, the control unit 1030 creates standard metadata of objects in the image that conforms to ARSEI, and updates ar_object_summary (7002) and ar_object_summary_canceled (7003).

[0176] In step S602, the control unit 1030 creates statistical information ar_object_summary (7002) and ar_object_summary_canceled (7003) about objects in the image with the ARSEI extension.

[0177] FIG. 13 is a flowchart showing the detailed procedure of step S602 in FIG.

[0178] In step S701, the control unit 1030 determines whether bit X corresponding to ar_object_idx[i](7001)==X in the statistical information ar_object_summary(7002) of objects present in the image is zero or non-zero. If bit X of ar_object_summary(7002) is non-zero, the process proceeds to step S702. If bit X of ar_object_summary(7002) is zero, the process proceeds to step S703.

[0179] In step S702, the control unit 1030 determines whether or not there is an object X in the image corresponding to ar_object_idx[i](7001)==X, based on the object detection information detected by the object detection unit 1060. If it is determined that there is an object X, the process proceeds to step S704, and if it is determined that there is no object X, the process proceeds to step S705.

[0180] In step S703, the control unit 1030 determines whether or not there is an object X in the image corresponding to ar_object_idx[i](7001)==X, based on the object detection information detected by the object detection unit 1060. If it is determined that there is an object X, the process proceeds to step S706, and if it is determined that there is no object X, the process in FIG. 13 ends.

[0181] In step S704, the control unit 1030 sets bit X corresponding to ar_object_idx[i] (7001)==X in ar_object_summary (7002) to 1 in order to update the object information of object X.

[0182] In step S707 following step S704, the control unit 1030 stores index X as metadata in the i-th position of the array of ar_object_idx (7001).

[0183] In step S708 following step S707, the control unit 1030 sets ar_object_cancel_flag (7006) to 0 and stores it in the metadata. Also in step S708, the control unit 1030 sets ar_bounding_box_update_flag (7007) to 1 and stores it in the metadata.

[0184] In step S709 following step S708, the control unit 1030 stores the position and size of object X in the metadata as bounding box information 7006. Note that the bounding box information 7008 indicating the position and size information of the object is described by ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]], as shown in Fig. 11 .

[0185] In step S705, the control unit 1030 clears to 0 the bit X corresponding to ar_object_idx[i] (7001)==X in the ar_object_summary (7002) in order to delete the object information of object X.

[0186] In step S710 following step S705, the control unit 1030 stores index X as metadata in the i-th position of the array of ar_object_idx (7001).

[0187] In step S711 following step S710, the control unit 1030 sets ar_object_cancel_flag (7006) to 1 and stores it in the metadata.

[0188] In step S712 following step S711, the control unit 1030 sets bit X corresponding to ar_object_idx[i] (7001)==X in ar_object_summary_canceled (7003) to 1.

[0189] In step S706, the control unit 1030 sets bit X corresponding to ar_object_idx[i] (7001)==X in ar_object_summary (7002) to 1 in order to create object information for object X.

[0190] In step S713 following step S706, the control unit 1030 stores index X as metadata in the i-th position of the array of ar_object_idx (7001).

[0191] In step S714 following step S713, the control unit 1030 sets ar_object_cancel_flag (7006) to 0, sets ar_object_label_update_flag (7009) to 1 and stores them as metadata.

[0192] In step S715 following step S714, the control unit 1030 stores the label number of object X that identifies ar_label (7005) in the metadata at the i-th position in the array of ar_object_label_idx (7010).

[0193] In step S716 following step S715, the control unit 1030 sets ar_bounding_box_update_flag (7007) to 1 and stores it in the metadata.

[0194] In step S717 following step S716, the control unit 1030 stores the position and size of object X as bounding box information 7008 in the metadata.

[0195] FIG. 14 is a flowchart showing the detailed procedure of step S603 in FIG.

[0196] In step S801, the control unit 1030 stores index 0 as metadata in the i-th position of the array of ar_object_idx (7001).

[0197] In step S802 following step S801, the control unit 1030 stores the ar_object_summary (7002) in steps S704 to S706 in FIG. 13 as metadata.

[0198] In step S803 following step S802, the control unit 1030 stores the ar_object_summary_canceled (7003) in step S712 of FIG. 13 as metadata.

[0199] In step S804 following step S803, the control unit 1030 searches bits 1 to 255 of ar_object_summary (7002) and ar_object_summary_canceled (7003). Then, the control unit 1030 creates ar_object_summary_label_idx[X] (7004), where X is an index from 1 to 255 excluding the objects that describe ar_object_summary (7002) and ar_object_summary_canceled (7003), which are index 0.

[0200] In step S805 following step S804, the control unit 1030 clears all bits of the ar_object_summary_canceled (7003) to 0.

[0201] FIG. 15 is a flowchart showing the detailed procedure of step S804 in FIG.

[0202] In step S901, the control unit 1030 determines whether bit X corresponding to index X of ar_object_summary (7002) is 0 or non-zero. If it is determined in step S901 that bit X is 0, the process in Fig. 15 ends. If it is determined in step S901 that bit X is non-zero, the process proceeds to step S902.

[0203] In step S902, the control unit 1030 determines whether bit X corresponding to index X of ar_object_summary_canceled (7003) is 0 or non-0. If it is determined in step S902 that bit X is 0, the process in Fig. 15 ends. If it is determined in step S902 that bit X is non-0, the process proceeds to step S903.

[0204] In step S903, the control unit 1030 updates the ar_object_summary_label_idx[X] (7004) corresponding to the index X to the latest label of the object X and stores this in the metadata.

[0205] Hereinafter, with reference to FIG. 16, a procedure for analyzing the ar_object_summary (7002) and ar_object_summary_canceled (7003) included in the metadata by the video receiving device 3000 in this embodiment will be described.

[0206] 16 is a flowchart showing an extended metadata analysis method of the video receiving device 3000. The control unit 1130 executes a program stored in the storage unit 1140 to realize the processing of the flowchart in FIG.

[0207] In step S1001, the control unit 1130 determines whether bit X corresponding to index X of ar_object_summary (7002) is non-zero or 0. If it is determined in step S1001 that bit X is non-zero, the process proceeds to step S1002. If it is determined in step S1001 that bit X is 0, the process proceeds to step S1003.

[0208] In step S1002, the control unit 1130 determines whether bit X corresponding to index X of ar_object_summary_canceled (7003) is non-zero or 0. If it is determined in step S1002 that bit X is non-zero, the process proceeds to step S1004. If it is determined in step S1002 that bit X is 0, the process in FIG. 16 ends.

[0209] In step S1003, the control unit 1130 determines whether or not there is an object X linked by index X in the object group managed by the video receiving device 3000. If it is determined in step S1003 that there is an object X, the process proceeds to step S1005. If it is determined in step S1003 that there is no object X, the process in FIG. 16 ends.

[0210] In step S1004, the control unit 1130 determines whether or not there is an object X linked by index X in the object group managed by the video receiving device 3000. If it is determined in step S1004 that there is an object X, the process proceeds to step S1006. If it is determined in step S1004 that there is no object X, the process proceeds to step S1007.

[0211] In step S1005, the control unit 1130 deletes object X. In the case where metadata in which ar_object_cancel_flag (7006) is 1 is missing, it is possible to correct the anomaly in which object X exists on the video receiving device 3000 side even though object X does not exist in the current image.

[0212] In step S1006, the control unit 1130 refers to ar_object_summary_label_idx[X] (7004) and updates the label of object X to the latest one. In cases where metadata where ar_object_cancel_flag (7006) is 1 and metadata where a new object appears at index X are missing, it is possible to correct the label that is likely to be incorrect and that is associated with object X on the video receiving device 3000 side.

[0213] In step S1007, the control unit 1130 creates a new object X in the case where the object X does not exist on the video receiving device 3000 side but exists in the current image.

[0214] In this embodiment, the object information of object X described together with ar_object_summary (7002) is only a label. The position, size, etc. are omitted because they are likely to be updated with standard objects. If an increase in the data amount of the object describing ar_object_summary (7002) is allowed, the position and size can be stored by storing bounding box information, etc., on the video transmitting device 1000 side. As the data amount increases, data becomes more likely to be lost on the communication path, so this is a section that can be changed depending on the application.

[0215] The second metadata SEI5 in Fig. 19 includes information ar_object_summary (7002) indicating the appearance of an object in a frame for each object index in steps S704 to S706 in Fig. 13, and information ar_object_summary_canceled (7003) indicating the disappearance of an object in a frame for each object index in step S712 in Fig. 13. These are stored (generated) as metadata in steps S802 and S803 in Fig. 14.

[0216] Furthermore, the second metadata SEI5 includes information ar_object_summary_label_idx[X] (7004) on the latest label of the object in the frame for each object index in step S903 of FIG.

[0217] Next, an example of a case where an error occurs in receiving the first metadata SEI1 indicating the disappearance of an object with a first label and the first metadata SEI2 indicating the appearance of an object with a second label will be described.

[0218] In this case, the control unit 1030 transmits first metadata SEI1 indicating the disappearance of an object corresponding to a first index to which a first label has been assigned, then transmits first metadata SEI2 indicating the appearance of an object corresponding to the first index to which a second label different from the first label has been assigned, and transmits second metadata SEI5 including information indicating the disappearance of the object at the first index, information indicating the appearance of the object at the first index, and information on the latest label of the object at the first index.

[0219] If the control unit 1130 does not receive first metadata SEI1 indicating the disappearance of an object corresponding to a first index to which a first label has been assigned, does not receive first metadata SEI2 indicating the appearance of an object corresponding to a first index to which a second label different from the first label has been assigned, but receives second metadata SEI5 including information indicating the disappearance of an object at the first index, information indicating the appearance of an object at the first index, and information on the latest label of the object at the first index, and if an object at the first index exists, the control unit 1130 proceeds from step S1004 to step S1006 in FIG. 16.

[0220] In step S1006, the control unit 1130 controls the object of the first index to be displayed with the latest label changed based on the latest label information ar_object_summary_label_idx[X] (7004).

[0221] Next, step S1007 in Fig. 16 will be described. When the control unit 1130 receives second metadata SEI5 including information indicating the disappearance of the object at the first index and information indicating the appearance of the object at the first index, and when the object at the first index does not exist, the process proceeds from step S1004 to step S1007 in Fig. 16. In step S1007, the control unit 1130 performs control so that information on the object at the first index is newly created and displayed.

[0222] Next, step S1005 in Fig. 16 will be described. When control unit 1130 receives second metadata that does not include information indicating the appearance of an object at the first index and when an object at the first index exists, control unit 1130 proceeds from step S1003 to step S1005 in Fig. 16. In step S1005, control unit 1130 performs control so that information about the object at the first index is not displayed.

[0223] In step S601 of FIG. 12, while an object appears, moves, or disappears within an image, the amount of metadata data increases, so insertion may be suppressed to reduce the amount of communication data.

[0224] That is, when the transmission frequency of the first metadata SEI1 to SEI4 in FIG. 19 is equal to or greater than the threshold, the control unit 1030 reduces the transmission frequency of the second metadata SEI5 compared to when the transmission frequency of the first metadata SEI1 to SEI4 is less than the threshold.

[0225] Although the preferred embodiments have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist thereof.

[0226] For example, the object storing the ar_object_summary (7002) has been set to ar_object_idx[i] (7001)==0, but it is also acceptable if the ar_object_label_idx[X] (7010) is invalid.

[0227] In this case, the first metadata SEI1 to SEI4 in Fig. 19 have information on the valid label of each object. The second metadata SEI5 is information on the invalid label of an object. When the control unit 1030 receives metadata in which the label of an object is information on a valid label, the control unit 1030 recognizes that the received metadata is the first metadata SEI1 to SEI4. When the control unit 1030 receives metadata in which the label of an object is information on an invalid label, the control unit 1030 recognizes that the received metadata is the second metadata SEI5.

[0228] Similarly, the ar_bounding_box_width[X] and ar_bounding_box_height[X] of the bounding box information 7008 may be invalid (0), or may be larger than the image size.

[0229] First, a case where the size of bounding box information 7008 is 0 will be described. First metadata SEI1 to SEI4 in FIG. 19 have information on the size of the bounding box surrounding each object. Second metadata SEI5 has a bounding box size of 0. When the control unit 1030 receives metadata in which the bounding box size is 1 or greater, it recognizes that the received metadata is first metadata SEI1 to SEI4. Furthermore, when the control unit 1030 receives metadata in which the bounding box size is 0, it recognizes that the received metadata is second metadata SEI5.

[0230] Next, a case where the size of the bounding box information 7008 is larger than the image size will be described. The first metadata SEI1 to SEI4 in FIG. 19 have information on the size of the bounding box surrounding each object. The second metadata SEI5 has a bounding box size larger than the frame image size. When the control unit 1030 receives metadata whose bounding box size is equal to or smaller than the frame image size, it recognizes the received metadata as the first metadata SEI1 to SEI4. Furthermore, when the control unit 1030 receives metadata whose bounding box size is larger than the frame image size, it recognizes the received metadata as the second metadata SEI5.

[0231] As described above, according to the first to third embodiments, it is possible to improve the robustness of communication channels regarding metadata in systems that adopt the H.265 standard "Annotated Region SEI (ARSEI)" and its successor standards.

[0232] The video receiving device 3000 can supplement the missing first metadata SEI3 and appropriately correct object information in the video stream by using the second metadata SEI5. Even if the loss of the first metadata SEI3 on ​​the video transmission path causes the loss of object appearance / disappearance information, resulting in erroneous object information, the video receiving device 3000 can appropriately correct the object by using the second metadata SEI5.

[0233] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0234] It should be noted that the above-described embodiments merely illustrate specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0235] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A video transmitting device; a video receiving device; The video transmission device a detection means for detecting an object included in each frame of a plurality of frame images; a first communication means for transmitting, when the detection means detects that an object has changed between frames, first metadata including information about the changed object to the video receiving device together with an image of the frame; the first communication means transmits, for a predetermined frame, second metadata including information about the object in the frame detected by the detection means, together with an image of the frame to the video receiving device; The video receiving device second communication means for receiving, for some frames, an image of the frame together with the first metadata, and for other frames, an image of the frame together with the second metadata; a display control means for controlling, when the first metadata is received, to update information of an object to be displayed based on the first metadata, and for controlling, when the second metadata is received, to display information of the object based on the second metadata; A video transmission / reception system comprising: (Configuration 2) The video transmitting and receiving system according to configuration 1, wherein the display control means, when receiving the first metadata, controls to update information of an object to be superimposed on the image of the received frame based on the first metadata, and when receiving the second metadata, controls to superimpose information of the object on the received frame based on the second metadata. (Configuration 3) 3. The video transmitting / receiving system according to configuration 1 or 2, wherein the second metadata includes information indicating the presence or absence of an object in a frame for each index of the object. (Configuration 4) the first communication means transmits first metadata indicating the appearance of a first object, then transmits first metadata indicating the disappearance of the first object, and then transmits second metadata indicating the absence of the first object; The video transmitting and receiving system according to configuration 3, wherein the display control means controls so as not to display information about the first object when, after receiving first metadata indicating the appearance of the first object, it does not receive first metadata indicating the disappearance of the first object but receives second metadata indicating the absence of the first object. (Configuration 5) 3. The video transmitting / receiving system according to configuration 1 or 2, wherein the second metadata includes, for each area in the frame, information indicating the presence or absence of an object in the frame for each object index. (Configuration 6) the first communication means transmits first metadata indicating that a position of a first object is in a first region within a frame, then transmits first metadata indicating movement of the first object, and transmits second metadata indicating that the first object is not in the first region and that the first object is present in a second region different from the first region; The video transmitting and receiving system according to configuration 5, wherein the display control means, after receiving first metadata indicating that the position of the first object is in a first region within a frame, does not receive first metadata indicating movement of the first object, and if it receives second metadata indicating that the first object is not in the first region and that the first object is present in a second region different from the first region, controls to display information about the first object in the second region rather than in the first region. (Configuration 7) 7. The video transmitting / receiving system according to configuration 6, wherein the display control means controls to display a warning in the area of ​​the first object. (Configuration 8) 3. The video transmitting and receiving system according to configuration 1 or 2, wherein the second metadata includes information indicating the appearance of an object in a frame for each object index, and information indicating the disappearance of an object in a frame for each object index. (Configuration 9) 9. The video transmitting / receiving system according to configuration 8, wherein the second metadata further includes information on the latest label of an object in a frame for each index of the object. (Configuration 10) the first communication means transmits first metadata indicating the disappearance of an object corresponding to a first index to which a first label has been assigned, and then transmits first metadata indicating the appearance of an object corresponding to the first index to which a second label different from the first label has been assigned, and transmits second metadata including information indicating the disappearance of the object at the first index, information indicating the appearance of the object at the first index, and information on the latest label of the object at the first index; The video transmitting and receiving system according to configuration 9, wherein the display control means receives second metadata including information indicating the disappearance of the object of the first index, information indicating the appearance of the object of the first index, and information on the latest label of the object of the first index, without receiving first metadata indicating the disappearance of an object corresponding to a first index to which a first label has been assigned, and without receiving first metadata indicating the appearance of an object corresponding to the first index to which a second label different from the first label has been assigned, and if the object of the first index exists, controls to change the label of the object of the first index to the latest label and display it. (Configuration 11) The video transmission / reception system of configuration 10, characterized in that the display control means receives second metadata including information indicating the disappearance of an object of a first index and information indicating the appearance of an object of the first index, and, if the object of the first index does not exist, controls to newly create and display information for the object of the first index. (Configuration 12) The video transmission / reception system according to configuration 10 or 11, characterized in that the display control means receives second metadata that does not include information indicating the appearance of an object of a first index, and, when the object of the first index exists, controls so as not to display information about the object of the first index. (Configuration 13) The video transmission / reception system according to any one of configurations 1 to 12, wherein the first communication means transmits the second metadata together with an image of an I frame, an image of a P frame immediately before the I frame, or an image of a P frame immediately after the I frame. (Configuration 14) The video transmission / reception system according to any one of configurations 1 to 13, wherein the first communication means, when the transmission frequency of the first metadata is equal to or greater than a threshold, transmits the second metadata less frequently than when the transmission frequency of the first metadata is less than the threshold. (Configuration 15) In the first metadata, each object is identified by an index having a value of 1 or more; The second metadata has an object index of 0, The video transmission / reception system according to any one of configurations 1 to 14, wherein the display control means, when receiving metadata in which an object index is a value of 1 or greater, recognizes the received metadata as the first metadata, and, when receiving metadata in which an object index is 0, recognizes the received metadata as the second metadata. (Configuration 16) The first metadata includes information of a label for which each object is valid; The second metadata is information on an invalid label of the object; The video transmission / reception system according to any one of configurations 1 to 14, wherein the display control means, when receiving metadata in which an object label is information on a valid label, recognizes the received metadata as the first metadata, and, when receiving metadata in which an object label is information on an invalid label, recognizes the received metadata as the second metadata. (Configuration 17) the first metadata includes information about the size of a bounding box surrounding each object; the second metadata indicates that the size of the bounding box is 0; The video transmission / reception system according to any one of configurations 1 to 14, wherein the display control means, when receiving metadata in which the size of the bounding box is 1 or more, recognizes the received metadata as the first metadata, and, when receiving metadata in which the size of the bounding box is 0, recognizes the received metadata as the second metadata. (Configuration 18) the first metadata includes information about the size of a bounding box surrounding each object; The second metadata includes information indicating that the size of the bounding box is larger than the size of the image of the frame; The video transmission / reception system according to any one of configurations 1 to 14, wherein the display control means, when receiving metadata in which the size of the bounding box is equal to or smaller than the size of the frame image, recognizes the received metadata as the first metadata, and, when receiving metadata in which the size of the bounding box is larger than the size of the frame image, recognizes the received metadata as the second metadata. (Configuration 19) a detection means for detecting an object included in each frame of a plurality of frame images; a communication means for transmitting, when the detection means detects that an object has changed between frames, first metadata including information about the changed object together with an image of the frame; The video transmission device is characterized in that the communication means transmits, for a predetermined frame, second metadata including information about an object in the frame detected by the detection means, together with an image of the frame. (Configuration 20) communication means for receiving, for some frames, images of the frames together with first metadata containing information of objects that have changed between frames, and for other frames, images of the frames together with second metadata containing information of objects in the frames; a display control means for controlling, when the first metadata is received, to update information of an object to be displayed based on the first metadata, and for controlling, when the second metadata is received, to display information of the object based on the second metadata; A video receiving device comprising: (Method 1) A video transmitting device; A processing method for a video transmission / reception system having a video receiving device, a detection step in which the video transmission device detects an object included in each frame of a plurality of frames of images; a first transmission step in which, when it is detected by the detection step that an object has changed between frames, the video transmission device transmits first metadata including information about the changed object together with an image of the frame to the video reception device; a second transmission step in which the video transmitting device transmits, for a predetermined frame, second metadata including information about the object in the frame detected by the detection step, together with an image of the frame to the video receiving device; a receiving step in which the video receiving device receives, for some frames, an image of the frame together with the first metadata, and for other frames, an image of the frame together with the second metadata; a display control step in which, when the video receiving device receives the first metadata, the video receiving device controls to update information of an object to be displayed based on the first metadata, and, when the video receiving device receives the second metadata, the video receiving device controls to display information of the object based on the second metadata; A processing method for a video transmission / reception system, comprising: (Method 2) a detection step of detecting an object included in each frame of a plurality of frame images; a first transmission step of transmitting, when the detection step detects that an object has changed between frames, first metadata including information about the changed object together with an image of the frame; a second transmitting step of transmitting, for a predetermined frame, second metadata including information about the object in the frame detected by the detecting step, together with the image of the frame; A processing method for a video transmission device, comprising: (Method 3) receiving an image of a frame together with first metadata for some frames, the first metadata including information of objects that have changed between frames, and receiving an image of a frame together with second metadata for other frames, the second metadata including information of objects in the frame; a display control step of, when the first metadata is received, controlling to update information of an object to be displayed based on the first metadata, and, when the second metadata is received, controlling to display information of the object based on the second metadata; A processing method for a video receiving device, comprising: (Program 1) a detection step of detecting an object included in each frame of a plurality of frame images; a first transmission step of transmitting, when the detection step detects that an object has changed between frames, first metadata including information about the changed object together with an image of the frame; a second transmitting step of transmitting, for a predetermined frame, second metadata including information about the object in the frame detected by the detecting step, together with the image of the frame; A program that causes a computer to execute the following. (Program 2) receiving an image of a frame together with first metadata for some frames, the first metadata including information of objects that have changed between frames, and receiving an image of a frame together with second metadata for other frames, the second metadata including information of objects in the frame; a display control step of, when the first metadata is received, controlling to update information of an object to be displayed based on the first metadata, and, when the second metadata is received, controlling to display information of the object based on the second metadata; A program that causes a computer to execute the following. [Explanation of symbols]

[0236] 1000 video transmission device, 1010 imaging unit, 1011 optical system, 1012 imaging element, 1013 development processing unit, 1020 communication unit, 1030 control unit, 1040 storage unit, 1050 compression encoding unit, 1060 object detection unit

Claims

1. A video transmitting device; a video receiving device; The video transmission device a detection means for detecting an object included in each frame of a plurality of frames of images; and first communication means for transmitting, when the detection means detects that an object has changed between frames, first metadata including information about the changed object to the video receiving device together with the image of the frame; the first communication means transmits, for a predetermined frame, second metadata including information about the object in the frame detected by the detection means, together with an image of the frame to the video receiving device; The video receiving device second communication means for receiving, for some frames, an image of the frame together with the first metadata, and for other frames, an image of the frame together with the second metadata; a display control means for controlling, when the first metadata is received, to update information of an object to be displayed based on the first metadata, and for controlling, when the second metadata is received, to display information of the object based on the second metadata; A video transmission / reception system comprising:

2. 2. The video transmission / reception system according to claim 1, wherein, when the display control means receives the first metadata, it controls to update information about an object to be superimposed on an image of the received frame based on the first metadata, and when the display control means receives the second metadata, it controls to superimpose information about the object on the received frame based on the second metadata.

3. 2. The video transmission / reception system according to claim 1, wherein the second metadata includes information indicating the presence or absence of an object in a frame for each index of the object.

4. the first communication means transmits first metadata indicating the appearance of a first object, then transmits first metadata indicating the disappearance of the first object, and then transmits second metadata indicating the absence of the first object; 4. The video transmitting and receiving system according to claim 3, wherein the display control means controls so as not to display information about the first object if, after receiving first metadata indicating the appearance of the first object, it does not receive first metadata indicating the disappearance of the first object but receives second metadata indicating the absence of the first object.

5. 2. The video transmission / reception system according to claim 1, wherein the second metadata includes information indicating the presence or absence of an object in the frame for each area in the frame and for each object index.

6. the first communication means transmits first metadata indicating that a position of a first object is in a first region within a frame, and then transmits first metadata indicating movement of the first object, and transmits second metadata indicating that the first object is not in the first region and that the first object is present in a second region different from the first region; 6. The video transmission and reception system according to claim 5, wherein, after receiving first metadata indicating that the position of the first object is in a first region within a frame, if the display control means does not receive first metadata indicating movement of the first object but receives second metadata indicating that the first object is not in the first region and that the first object is present in a second region different from the first region, the display control means controls to display information about the first object in the second region rather than in the first region.

7. 7. The video transmitting and receiving system according to claim 6, wherein the display control means controls the display so as to display a warning in the area of ​​the first object.

8. 2. The video transmission / reception system according to claim 1, wherein the second metadata includes information indicating the appearance of an object in a frame for each object index, and information indicating the disappearance of an object in a frame for each object index.

9. 9. The video transmitting and receiving system according to claim 8, wherein the second metadata further includes information on the latest label of an object in a frame for each index of the object.

10. the first communication means transmits first metadata indicating the disappearance of an object corresponding to a first index to which a first label has been assigned, and then transmits first metadata indicating the appearance of an object corresponding to the first index to which a second label different from the first label has been assigned, and transmits second metadata including information indicating the disappearance of the object at the first index, information indicating the appearance of the object at the first index, and information on the latest label of the object at the first index; 10. The video transmission and reception system according to claim 9, wherein the display control means receives second metadata including information indicating the disappearance of an object corresponding to a first index to which a first label has been assigned, information indicating the appearance of an object corresponding to the first index to which a second label different from the first label has been assigned, and information on a latest label of the object corresponding to the first index, and, if the object corresponding to the first index exists, controls to change the label of the object corresponding to the first index to the latest label and display it.

11. The video transmission and reception system according to claim 10, characterized in that the display control means receives second metadata including information indicating the disappearance of an object of a first index and information indicating the appearance of an object of the first index, and, if the object of the first index does not exist, controls to newly create and display information of the object of the first index.

12. The video transmission and reception system of claim 10, characterized in that the display control means receives second metadata that does not include information indicating the appearance of an object with a first index, and, when an object with the first index exists, controls so as not to display information about the object with the first index.

13. 2. The video transmission / reception system according to claim 1, wherein the first communication means transmits the second metadata together with an image of an I frame, an image of a P frame immediately before the I frame, or an image of a P frame immediately after the I frame.

14. 2. The video transmission / reception system according to claim 1, wherein the first communication means, when the transmission frequency of the first metadata is equal to or greater than a threshold, transmits the second metadata less frequently than when the transmission frequency of the first metadata is less than the threshold.

15. In the first metadata, each object is identified by an index of one or more values; The second metadata has an object index of 0, The video transmission / reception system according to claim 1, characterized in that, when the display control means receives metadata in which an object index is a value of 1 or greater, it recognizes the received metadata as the first metadata, and, when the display control means receives metadata in which an object index is 0, it recognizes the received metadata as the second metadata.

16. The first metadata includes information of a label for which each object is valid; the second metadata is information on an invalid label of the object; 2. The video transmission / reception system according to claim 1, wherein, when the display control means receives metadata in which an object label is information about a valid label, the display control means recognizes the received metadata as the first metadata, and when the display control means receives metadata in which an object label is information about an invalid label, the display control means recognizes the received metadata as the second metadata.

17. the first metadata includes information about the size of a bounding box surrounding each object; The second metadata indicates that the size of the bounding box is 0; The video transmission and reception system according to claim 1, characterized in that, when the display control means receives metadata in which the size of the bounding box is 1 or more, it recognizes the received metadata as the first metadata, and, when the display control means receives metadata in which the size of the bounding box is 0, it recognizes the received metadata as the second metadata.

18. the first metadata includes information about the size of a bounding box surrounding each object; The second metadata includes information indicating that the size of the bounding box is greater than the size of an image of the frame; 2. The video transmission and reception system according to claim 1, wherein, when the display control means receives metadata whose bounding box size is equal to or smaller than the size of the frame image, the display control means recognizes the received metadata as the first metadata, and when the display control means receives metadata whose bounding box size is larger than the size of the frame image, the display control means recognizes the received metadata as the second metadata.

19. a detection means for detecting an object included in each frame of a plurality of frames of images; a communication means for transmitting, when the detection means detects that an object has changed between frames, first metadata including information about the changed object together with an image of the frame; The video transmitting device is characterized in that the communication means transmits, for a predetermined frame, second metadata including information about an object in the frame detected by the detection means, together with an image of the frame.

20. a communication means for receiving, for some frames, images of the frames together with first metadata containing information of objects that have changed between frames, and for other frames, images of the frames together with second metadata containing information of objects in the frames; a display control means for controlling, when the first metadata is received, to update information of an object to be displayed based on the first metadata, and for controlling, when the second metadata is received, to display information of the object based on the second metadata; A video receiving device comprising:

21. A video transmitting device; A processing method for a video transmission / reception system having a video receiving device, a detection step in which the video transmission device detects an object included in each frame of a plurality of frames of images; a first transmission step in which, when it is detected by the detection step that an object has changed between frames, the video transmission device transmits first metadata including information about the changed object together with an image of the frame to the video reception device; a second transmission step in which the video transmitting device transmits, for a predetermined frame, second metadata including information about the object in the frame detected by the detection step, together with an image of the frame to the video receiving device; a receiving step in which the video receiving device receives, for some frames, an image of the frame together with the first metadata, and for other frames, an image of the frame together with the second metadata; a display control step in which, when the video receiving device receives the first metadata, the video receiving device controls to update information of an object to be displayed based on the first metadata, and, when the video receiving device receives the second metadata, the video receiving device controls to display information of the object based on the second metadata; A processing method for a video transmission / reception system, comprising:

22. a detection step of detecting an object included in each frame of a plurality of frame images; a first transmission step of transmitting, when the detection step detects that an object has changed between frames, first metadata including information about the changed object together with an image of the frame; a second transmitting step of transmitting, for a predetermined frame, second metadata including information about the object in the frame detected by the detecting step, together with the image of the frame; A processing method for a video transmission device, comprising:

23. receiving an image of a frame together with first metadata for some frames, the first metadata including information of objects that have changed between frames, and receiving an image of a frame together with second metadata for other frames, the second metadata including information of objects in the frame; a display control step of, when the first metadata is received, controlling to update information of an object to be displayed based on the first metadata, and, when the second metadata is received, controlling to display information of the object based on the second metadata; A processing method for a video receiving device, comprising:

24. a detection step of detecting an object included in each frame of a plurality of frame images; a first transmission step of transmitting, when the detection step detects that an object has changed between frames, first metadata including information about the changed object together with an image of the frame; a second transmitting step of transmitting, for a predetermined frame, second metadata including information about the object in the frame detected by the detecting step, together with the image of the frame; A program that causes a computer to execute the following.

25. receiving an image of a frame together with first metadata for some frames, the first metadata including information of objects that have changed between frames, and receiving an image of a frame together with second metadata for other frames, the second metadata including information of objects in the frame; a display control step of, when the first metadata is received, controlling to update information of an object to be displayed based on the first metadata, and, when the second metadata is received, controlling to display information of the object based on the second metadata; A program that causes a computer to execute the following.

Images