Information processing system, server device and program

Abstract

To allow for more efficient identification of data having a desired timing.SOLUTION: An information processing system includes a server device and edge systems. The server device includes a data receiving unit, a reference information receiving unit, a request processing unit, and an identifying unit. The data receiving unit receives multiple pieces of first stream data to which first time information is associated from the edge systems. The reference information receiving unit receives reference information in which second time information and the first time information corresponding to the selected first stream data are associated to each other, from the edge systems. The request processing unit receives an acquisition request for the first stream data corresponding to third time information represented in the same format as the second time information. The identifying unit identifies the first stream data to which the first time information corresponding to the third time information is associated based on the reference information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing system, a server device, and a program. [Background technology]

[0002] In recent years, in cloud services that collect and distribute video, there has been an increasing need for higher functionality due to the sophistication of data processing, such as when playing back video from a certain point in time in the past indicated by time-series data, or when extracting and saving video from a certain point in time in the past as a file. Time-series data is, for example, data detected by a temperature sensor. Time-series data is associated with, for example, time information (year, month, day, hour, minute, second, etc.) indicating the timing of detection. Similarly, video stream data (video stream data) is associated with, for example, time information (frame number, etc.) indicating the timing of playback.

[0003] To meet these needs, a mechanism is needed that links the time information of the time series data with the time information of the video stream data, allowing the cloud service to identify which time information in the video stream data corresponds to the desired playback position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-219725 [Patent Document 2] Japanese Patent Application Publication No. 2018-151918 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide an information processing system, a server device, and a program that enable more efficient identification of data at a desired timing. [Means for solving the problem]

[0006] An information processing system according to an embodiment includes a server device and an edge system. The server device includes a data receiving unit, a reference information receiving unit, a request processing unit, and an identifying unit. The data receiving unit receives, from the edge system, a plurality of first stream data items associated with first time information. The reference information receiving unit receives, from the edge system, reference information that associates second time information with the first time information corresponding to selected first stream data items. The request processing unit receives an acquisition request for the first stream data items associated with third time information that is expressed in the same format as the second time information. The identifying unit identifies the first stream data items associated with the first time information that corresponds to the third time information based on the reference information. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of an information processing system according to a first embodiment. [Figure 2] FIG. 4 is a diagram showing an example of a data structure of reference information. [Figure 3] FIG. 3 is a sequence diagram of information processing according to the first embodiment. [Figure 4] 10 is a flowchart of a reference information transmission process according to the first embodiment. [Figure 5] FIG. 10 is a block diagram of an information processing system according to a first modified example. [Figure 6] FIG. 10 is a diagram showing an example of a data structure of reference information according to the first modification. [Figure 7] FIG. 10 is a block diagram of an information processing system according to a second modification. [Figure 8] FIG. 13 is a block diagram of an information processing system according to a fourth modification. [Figure 9] FIG. 13 is a sequence diagram of information processing in Modification 4. [Figure 10] 13 is a flowchart of a reference information transmission process in Modification 4. [Figure 11] FIG. 10 is a block diagram of an information processing system according to a second embodiment. [Figure 12] 10 is a flowchart of a reference information transmission process according to the second embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a data structure of reference information according to a first modified example of the second embodiment. [Figure 14] FIG. 10 is a diagram showing an example of a data structure of reference information according to a second modification of the second embodiment. [Figure 15] FIG. 10 is a block diagram of an information processing system according to a third embodiment. [Figure 16] FIG. 1 is a hardware configuration diagram of an apparatus according to first to third embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of an information processing system, a server device, and a program according to the present invention will be described in detail below with reference to the accompanying drawings.

[0009] In the following, an example will be described in which stream data that can be identified by time information indicating timing is processed. Stream data is, for example, data that is output over time. Stream data includes, for example, time-series data, which is a series of data that is generated at regular intervals and includes information indicating the time of generation.

[0010] The stream data may be any type of data, for example, sensor data output by a sensor (detection device) that detects a predetermined physical quantity at regular intervals. The sensor may be, for example, the following device, but may also be any other type of device. Camera (imaging device) Microphone (sound collection device) Temperature sensor Humidity sensor Radio wave sensor Sensors for measuring motor rotation speed, voltage, power, etc.

[0011] The stream data generated by such sensors is, for example, video stream data, audio stream data, and waveform data that indicates changes in the detection results.

[0012] The time information may be in any format as long as it is information that can identify the timing (temporal position) of the stream data. For example, the time information is expressed in the following format: (T1) Absolute time expressed based on a specified time: time according to the ISO (International Organization for Standardization) 8601 standard, Unix time, etc. (T2) Relative time expressed based on a freely definable timing: a frame number of video stream data, a time based on the start timing of a predetermined process or function, a data amount, or a data position.

[0013] The data position is a position that has a special meaning, such as the first byte or the 1001st byte. The data position may also be a position that has a special meaning, such as before or after data that separates data (or frames), expressed in a specific data format, such as an H.264 start code.

[0014] The time information may be in a format that expresses time as a number conforming to a predetermined format such as a timestamp based on the state of the clock inside the device, or may be in a format that expresses time in text format.

[0015] As a technique for identifying stream data, a technique for embedding a timestamp in the stream data to be identified, and a technique for controlling the processing of stream data based on the timing indicated by a specific counter have been proposed.

[0016] However, to apply these technologies to systems that handle stream data that does not contain time information such as timestamps, it is necessary to modify the data formats and processing procedures defined by the communication protocols used in those systems so that they can handle time information. Protocol modifications can be broadly divided into two types. One is to modify the specifications of the existing protocol itself to create a new, unique protocol. The other is to utilize optional parts of the existing protocol (data formats or processing that can be added at the user's or application's discretion). In either case, the modified protocol must be supported throughout the entire end-to-end system, from the edge system (edge ​​device) that generates the stream data to the cloud server (server equipment that provides cloud services).

[0017] However, modifying the entire system program is difficult. In particular, when handling stream data such as waveform data with a sampling rate of several hundred kHz or more, modifications such as adding time information such as timestamps to stream data samples or changing the processing timing of the stream data increase the amount of data handled and the processing load, making them difficult to implement.

[0018] Therefore, in the following embodiment, the edge system transmits reference information to the server for use in identifying the stream data, independently of the transmission of the stream data. The reference information is information that associates time information (such as a frame number) of the reference stream data with time information (such as a timestamp) in a format used to identify the stream data. When a user requests the acquisition of stream data, the server refers to the reference information to identify the time position of the requested stream data and returns the identified result to the user. This makes it possible to identify the desired stream data more efficiently.

[0019] Furthermore, as described above, the reference information is generated, transmitted, and stored independently of the function for processing stream data. Therefore, when applying this embodiment to an existing system, there is no need to modify the entire end-to-end of the existing system. In order to generate time information for stream data, it may be necessary to modify the program on the edge system side of the existing system. Even in this case, there is no need to modify the program on the cloud server side that collects the stream data.

[0020] Furthermore, the reference information can be generated at a granularity greater than the stream data (for example, one sample per second) that the user desires. Therefore, even when handling stream data with a sampling rate of several hundred KHz or more, it is not necessary to generate reference information for all samples. In other words, the amount of data and the processing load can be reduced. In this way, a system that can more efficiently identify desired stream data while satisfying the user's needs can be easily implemented.

[0021] (First embodiment) 1 is a block diagram showing an example of the configuration of an information processing system 10 according to the first embodiment. As shown in Fig. 1, the information processing system 10 has an edge system 100 and a server 200 (an example of a server device) connected via a network 500.

[0022] The network 500 is, for example, but not limited to, the Internet or a local area network (LAN), and may be a wired network, a wireless network, or a mixed wired and wireless network.

[0023] The server 200 is connected to a user terminal 400 used by a user requesting stream data, for example, via a network 500. A plurality of user terminals 400 may be connected to the server 200. The user terminal 400 is, for example, a personal computer or a mobile terminal.

[0024] The edge system 100 is connected to a camera 301 that generates video stream data. The video stream data is an example of a plurality of stream data (first stream data) associated with time information (first time information). The time information is, for example, a frame number. The edge system 100 and the camera 301 are connected via, for example, a control bus or a network 500. The camera 301 may be provided inside the edge system 100.

[0025] The server 200 may be physically configured by a single piece of hardware, or may be constructed as a cloud server in a cloud environment. The edge system 100 is, for example, physically configured by a single piece of hardware. An example in which an edge system is physically configured by multiple pieces of hardware (devices) will be described in the third embodiment.

[0026] 1 illustrates only one edge system 100 and one user terminal 400, the information processing system 10 may include multiple edge systems 100 and multiple user terminals 400. Also, as will be described in the following modified example, multiple cameras 301 may be included.

[0027] The edge system 100 includes a video transmission unit 101 (an example of a first data transmission unit), a generation unit 102, a reference information transmission unit 103, and a storage unit 121.

[0028] Video transmission unit 101 encodes the video stream data received from camera 301 and transmits the encoded video stream data to server 200. Video transmission unit 101 transmits the video stream data to server 200 while storing the operating status of video transmission unit 101 in storage unit 121 or the like as needed.

[0029] The operating status is the status of operations related to the encoding and transmission of video stream data. The operating status includes at least time information of the video stream data. The time information included in the operating status is, for example, the frame number of the last transmitted video stream data and the elapsed time since the video transmission unit 101 started operating. The generating unit 102 refers to the operating status when generating reference information.

[0030] The functions corresponding to the video transmission unit 101 and the video reception unit 201 and video storage unit 221 of the server 200 described below can be interpreted as corresponding to a system that collects stream data. When applying an existing system to this system, if the component corresponding to the video transmission unit 101 of the existing system does not have the function of storing the operating status, modification to add this function may be necessary. However, even in this case, modification such as adding a function is not required on the server 200 side.

[0031] The generating unit 102 generates the reference information when the conditions for generating the reference information (generation conditions) are satisfied.

[0032] The reference information is information that associates time information corresponding to video stream data selected as a reference from among multiple video stream data with time information (second time information) whose generation source is different from that of the time information of the video stream data. The time information of the video stream data will be referred to as time information TA below. The time information whose generation source is different from that of the time information of the video stream data will be referred to as time information TB below.

[0033] The time information TA and the time information TB may be any time information as long as they are generated from different sources. The time information TA and the time information TB may have different formats (data formats, display formats) or the same format. In the following, an example will be described in which the time information TA is a frame number, which is an example of relative time, and the time information TB is absolute time. In other words, an example will be described in which the time information TA and the time information TB have different formats.

[0034] As described above, even if the time information TA and the time information TB have the same format, the same procedure can be applied as long as they are generated from different sources. For example, the time information TA obtained from the internal clock of the temperature sensor and the time information TB obtained from the internal clock of the edge system 100 may be out of sync even if they have the same representation format (data format). According to this embodiment, by associating these two out-of-sync time information pieces that have the same format, it is possible to identify data at the desired timing.

[0035] The time information TA is obtained, for example, from the time information included in the operating status stored by the video transmission unit 101. When the frame number of the last transmitted video stream data is stored as the time information, the last transmitted video stream data by the video transmission unit 101 corresponds to the video stream data selected as the reference. The time information TB is obtained, for example, from a clock inside the edge system 100.

[0036] The conditions for generating the reference information are, for example, the following conditions. Generate baseline information at regular intervals (once per second, once per minute, etc.). Generates reference information when a control instruction is received from inside or outside the edge system 100.

[0037] The control instruction may be output by the video transmission unit 101, for example, each time the transmission of one frame of video stream data is completed, or may be output by the video transmission unit 101 each time the transmission of a specific frame in the video stream data is completed. A "specific frame in the video stream data" is, for example, a key frame (encoded data in the video stream data that can be decoded by itself) or a frame (first frame) corresponding to the beginning of a frame count (a frame number within a second that goes around once in one second).

[0038] The generating unit 102 generates the reference information when the generation condition is satisfied. The generating unit 102 may also perform calibration based on measurements of delay times and the like required for internal processing, and correct the time information used as the reference information. The calibration may be performed only once after the edge system 100 is started, or may be performed periodically after the edge system 100 is started. The generating unit 102 sends the generated reference information to the reference information transmitting unit 103.

[0039] The reference information transmission unit 103 transmits the reference information received from the generation unit 102 to the server 200. If an abnormality occurs during generation or transmission of the reference information, the generation unit 102 may re-execute the process of generating the reference information or the reference information transmission unit 103 may re-execute the process of transmitting the reference information, or the stream data in which the abnormality occurred may be discarded.

[0040] Each of the above units (video transmitting unit 101, generating unit 102, and reference information transmitting unit 103) is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0041] The storage unit 121 stores various data used in various processes of the edge system 100. For example, the storage unit 121 functions as a buffer for storing the operating status (time information) of the video transmission unit 101. Note that if the video transmission unit 101 includes a buffer inside, the storage unit 121 may not be provided.

[0042] The storage unit 121 can be configured from any commonly used storage medium such as a flash memory, a memory card, a RAM (Random Access Memory), an HDD (Hard Disk Drive), and an optical disk.

[0043] The server 200 includes a video receiving unit 201 (an example of a data receiving unit), a reference information receiving unit 202, a request processing unit 203, an identification unit 204, a video storage unit 221 (an example of a data storage unit), and a reference information storage unit 222.

[0044] The video receiving unit 201 receives video stream data from the edge system 100 and stores the received video stream data in the video storage unit 221. The video receiving unit 201 may perform preprocessing such as converting the video stream data into a format suitable for storage in the video storage unit 221. The preprocessing is, for example, processing such as converting the file format of the video stream data and transcoding the video stream data.

[0045] The video storage unit 221 stores the video stream data received from the edge system 100.

[0046] The reference information receiving unit 202 receives the reference information transmitted by the edge system 100 and stores the received reference information in the reference information storage unit 222 .

[0047] The reference information storage unit 222 stores (accumulates) reference information received from the edge system 100. FIG. 2 is a diagram showing an example of the data structure of reference information stored in the reference information storage unit 222. As shown in FIG. 2, the reference information includes identification information, time information TA, and time information TB. The identification information is, for example, information that identifies the edge system 100 that transmitted the reference information. In cases such as when the information processing system 10 includes one edge system 100, the reference information does not need to include identification information.

[0048] 1, the request processor 203 receives acquisition requests from users, applications, etc., and transmits responses to the acquisition requests. For example, the request processor 203 receives an acquisition request from the user terminal 400, etc., requesting the identification of video stream data, including a specification of time information (an example of third time information, hereinafter referred to as time information TC) expressed in the same format as the time information TB. Upon receiving the acquisition request, the request processor 203 instructs the identification unit 204 to identify the video stream data corresponding to the time information TC included in the acquisition request.

[0049] In addition, the request processing unit 203 transmits information indicating the video stream data identified by the identification unit 204 as data corresponding to the time information TC (such as the video stream data itself or identification information of the video stream data) to the sender of the acquisition request as a response to the acquisition request.

[0050] The identification unit 204 executes an identification process to identify video stream data corresponding to the time information TC included in the acquisition request. For example, the identification unit 204 refers to the reference information stored in the reference information storage unit 222 to identify video stream data associated with the time information TB corresponding to the time information TC included in the acquisition request.

[0051] The time information TB corresponding to the time information TC may be time information TB that matches the time information TC, or it may be time information TB whose difference from the time information TC is smaller than that of other time information (for example, time information TB with the smallest difference).

[0052] The identification unit 204 may obtain time information TB whose difference from the time information TC is smaller than that of other time information, and identify video stream data corresponding to a time shifted by an amount equivalent to the difference between the time information TB and the time information TC. In this case, the identification unit 204 may identify the video stream data by referring to parameters, meta information, etc. added to the video stream data stored in the video storage unit 221.

[0053] For example, if a frame rate is added to the video stream data, the identifying unit 204 uses the frame rate (e.g., FPS: frames per second) to determine the number of frames within a certain period of time (e.g., 1 second). The identifying unit 204 calculates the number of frames corresponding to the difference between time information TB and time information TC. The identifying unit 204 identifies the video stream data corresponding to a time that is shifted by the calculated number of frames from the determined time information TB as the video stream data corresponding to the time information TC.

[0054] Here, a specific example of the identification process by the identification unit 204 will be described. In the following, an example will be described in which the time information TA is the "frame number," the time information TB is the "absolute time of the edge system 100," and the time information TC specified in the acquisition request is the "absolute time of the edge system 100."

[0055] In this case, if there is reference information including time information TB that matches time information TC (T_req) among the reference information stored in reference information storage unit 222, identification unit 204 outputs the frame number (N_req) corresponding to T_req as the identification result. When video stream data can be identified by referring only to the reference information stored in reference information storage unit 222 as in this example, identification unit 204 does not need to refer to the video stream data in video storage unit 221, etc.

[0056] The identification result does not need to be a frame number, but may be the video stream data itself. For example, the identification unit 204 may read video stream data including the N_reqth frame from the video stream data stored in the video storage unit 221 and output the read video stream data as the identification result. The identification unit 204 may also output identification information such as a uniform resource locator (URL) for accessing the video stream data including the N_reqth frame as the identification result.

[0057] If there is no reference information stored in the reference information storage unit 222 that includes time information TB that matches the time information TC (T_req), the identification unit 204 identifies the video stream data using the reference information of the time information (T_req') that is closest to T_req.

[0058] For example, the identification unit 204 outputs the frame number corresponding to T_req' as the identification result (approximate solution). As in the above, the identification unit 204 may output the video stream data itself as the identification result instead of the frame number.

[0059] The identification unit 204 may generate new video stream data corresponding to time information TB that matches T_req from the video stream data read from the video storage unit 221, and output the generated video stream data as the identification result. For example, assume that video stream data starting from a certain time T0 is stored in the video storage unit 221, and a request is made to acquire video stream data starting from a time T1 that is later than time T0. In this case, the identification unit 204 generates new video stream data by deleting frames corresponding to times T0 to T1 from the video stream data starting from time T0, and outputs the generated video stream data as the identification result.

[0060] Each of the above units (video receiving unit 201, reference information receiving unit 202, request processing unit 203, and identification unit 204) is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0061] The video storage unit 221 and the reference information storage unit 222 can be configured using any commonly used storage medium, such as a flash memory, a memory card, a RAM, a HDD, an optical disc, etc. The video storage unit 221 and the reference information storage unit 222 may be configured using physically different storage media, or may be configured as different storage areas of the same physically different storage medium.

[0062] Next, a description will be given of information processing by the information processing system 10 according to the first embodiment configured as described above. Fig. 3 is a sequence diagram showing an example of information processing in the first embodiment.

[0063] The information processing by the information processing system 10 includes three processes: a process of transmitting and storing (accumulating) video stream data (steps S101 to S103), a process of transmitting and storing (accumulating) reference information (steps S104 to S106), and a process of identifying requested video stream data from the stored video stream data (steps S107 to S111). Although these three processes are shown consecutively in Fig. 3, each process can be executed independently.

[0064] When video stream data is received from camera 301, video transmission unit 101 encodes the received video stream data (step S101). Video transmission unit 101 transmits the encoded video stream data to server 200 (step S102).

[0065] The video receiving unit 201 of the server 200 converts the video stream data received from the edge system 100 (video transmitting unit 101), and stores the converted video stream data in the video storage unit 221 (step S103).

[0066] On the other hand, if the generation condition is satisfied, the generation unit 102 generates reference information (step S104). For example, if the generation condition is set to generate reference information every second, the generation unit 102 generates reference information every second. The reference information transmission unit 103 transmits the generated reference information to the server 200 (step S105).

[0067] The reference information receiving unit 202 of the server 200 stores the reference information received from the edge system 100 (reference information transmitting unit 103) in the reference information storage unit 222 (step S106).

[0068] Through the above-described processing, the video stream data and the reference information corresponding to the video stream data are stored (accumulated) in the server 200. Acquisition of the stored video stream data can be requested by specifying time information TC in the same format as the time information (time information TB) used in the reference information.

[0069] For example, when a request to acquire video stream data including time information TC is transmitted from user terminal 400, request processor 203 of server 200 receives the acquisition request (step S107). Request processor 203 requests identification unit 204 to identify the video stream data requested to be acquired (step S108).

[0070] The identifying unit 204 identifies the video stream data corresponding to the time information TC included in the acquisition request by referring to the reference information stored in the reference information storage unit 222 (step S109). The identifying unit 204 transmits information indicating the identified video stream data to the request processing unit 203 (step S110). The request processing unit 203 transmits a response including information indicating the video stream data to the user terminal 400 that sent the acquisition request (step S111).

[0071] Next, a description will be given of details of a reference information transmission process in which the edge system 100 generates and transmits reference information. The reference information transmission process corresponds to steps S104 and S105 in Fig. 3. Fig. 4 is a flowchart showing an example of the reference information transmission process.

[0072] The generating unit 102 determines whether the generation condition of the reference information is satisfied (step S201). If the generation condition of the reference information is not satisfied (step S201: No), the process returns to step S201 and repeats the process until the generation condition is satisfied.

[0073] If the conditions for generating reference information are satisfied (step S201: Yes), the generating unit 102 acquires time information TA from the operating status stored by the video transmitting unit 101 in the storage unit 121 or the like, and acquires time information TB from the clock inside the edge system 100 (step S202). The generating unit 102 generates reference information that associates the acquired time information TA and time information TB (step S203).

[0074] The reference information transmitting unit 103 transmits the generated reference information to the server 200 (step S204), and ends the reference information transmitting process.

[0075] (Variation 1) The number of cameras connected to one edge system 100 is not limited to one. Two or more cameras may be connected to the edge system 100. In Variation 1, an information processing system 10B including an edge system 100B to which two cameras 301 and 301B are connected will be described.

[0076] Fig. 5 is a block diagram showing an example of the configuration of an information processing system 10B according to Modification 1. As shown in Fig. 5, the information processing system 10B has a configuration in which an edge system 100B and a server 200B are connected via a network 500.

[0077] This modification differs from the above embodiment in that a camera 301B is further connected to the edge system 100B, a video transmission unit 101B is added to the edge system 100B, and a video reception unit 201B and a video storage unit 221B are added to the server 200B. The other configurations and functions are the same as those in FIG. 1, which is the block diagram of the information processing system 10 according to the first embodiment, so the same reference numerals are used and the description here will be omitted.

[0078] 5, edge system 100B includes multiple video transmitters 101 and 101B (multiple first data transmitters) that transmit video stream data captured by different cameras 301 and 301B to server 200B. Video transmitter 101B has the same functions as video transmitter 101, except that it processes and transmits video stream data received from camera 301B instead of camera 301.

[0079] Video receiving section 201B and video storage section 221B also have the same functions as video receiving section 201 and video storage section 221, except that the target data is replaced with video stream data received from camera 301B.

[0080] This modification can be applied to a system configuration that independently manages video stream data obtained from multiple cameras. For example, this modification can be applied to an information processing system in which camera 301 and camera 301B are a visible light camera and an infrared camera, respectively. Furthermore, this modification can be applied to an information processing system in which edge system 100B is a vehicle, and camera 301 and camera 301B are a front camera that captures images in front of the vehicle and a rear camera that captures images behind the vehicle, respectively.

[0081] Fig. 6 is a diagram showing an example of the data structure of the reference information in this modification. As shown in Fig. 6, the reference information includes identification information, time information TA1, time information TA2, and time information TB. The time information TA1 represents time information corresponding to the video stream data obtained from camera 301. The time information TA2 represents time information corresponding to the video stream data obtained from camera 301B.

[0082] By using the reference information as shown in FIG. 6, the identification unit 204 can identify data at a desired timing for data stored in both the video storage unit 221 and the video storage unit 221B.

[0083] In the first modification, reference information that associates time information of multiple pieces of video stream data can be generated, which makes it possible to more efficiently identify one piece of video stream data that corresponds to the other piece of video stream data at a desired timing.

[0084] For example, when a user checks the video stream data from a rear camera (e.g., camera 301B) and finds a following vehicle driving recklessly, the user requests video stream data from a front camera (e.g., camera 301) at the same timing. The identification unit 204 refers to the reference information and identifies the video stream data from the front camera corresponding to the relevant timing from the video storage unit 221. The request processing unit 203 provides the identified video stream data to the user.

[0085] (Variation 2) The second modification is an example of an information processing system 10C configured to synthesize video stream data from a plurality of connected cameras and store the synthesized data in a server.

[0086] Fig. 7 is a block diagram showing an example of the configuration of an information processing system 10C according to Modification 2. As shown in Fig. 7, the information processing system 10C has a configuration in which an edge system 100C and a server 200 are connected via a network 500.

[0087] This modification differs from the above embodiment in that a camera 301B is further connected to the edge system 100C and in the function of the video transmission unit 101C. The other configurations and functions are the same as those in FIG. 1, which is a block diagram of the information processing system 10 according to the first embodiment, so the same reference numerals are used and the description thereof will be omitted here.

[0088] Video transmission unit 101C differs from video transmission unit 101 of the above embodiment in that video transmission unit 101C combines video stream data captured by cameras 301 and 301B, which are different imaging devices, and transmits the combined video stream data to server 200. Note that, as in Modification 1, video transmission unit 101B may transmit video stream data captured by camera 301B to video receiving unit 201, and video receiving unit 201 may combine the video stream data.

[0089] This modification can be applied to an information processing system that transmits video stream data (such as a panoramic image) that combines video stream data from a front camera and a rear camera, for example.

[0090] (Variation 3) As described above, the stream data is not limited to video stream data. For example, when audio stream data is used as the stream data, a microphone, an audio transmitter (another example of the first data transmitter), an audio receiver (another example of the data receiver), and an audio memory (another example of the data memory) may be used instead of camera 301, video transmitter 101, video receiver 201, and video memory 221.

[0091] As another example, the camera 301 may be replaced with a sensor that handles uncompressed waveform data at a sampling rate of several hundred KHz or more, and the stream data that it handles may be replaced with waveform data. In the case of stream data in which the amount of data per sample is fixed, such as uncompressed waveform data, the time information TA of the stream data may be expressed as a relative time as follows: Number of bytes (counted from the beginning) Number of samples (counted from the beginning)

[0092] For example, if there is no time information TB that matches the time information TC, the identifying unit 204 finds time information TB whose difference from the time information TC is smaller than that of other time information. The identifying unit 204 calculates the number of samples corresponding to the difference between the found time information TB and the time information TC based on the sampling rate, the amount of data per sample, etc. The identifying unit 204 identifies the stream data corresponding to a time that is shifted from the time information TB by the calculated number of samples as the stream data corresponding to the time information TC.

[0093] In this way, in the first embodiment, the requested stream data is identified using the reference information transmitted from the edge system independently of the transmission of the stream data, which makes it possible to more efficiently identify the desired stream data.

[0094] (Variation 4) The function of the generation unit 102 may be divided into a function (instruction unit) that instructs the generation of reference information when the generation condition is satisfied, and a function that generates the reference information in response to the instruction. Fig. 8 is a block diagram showing an example configuration of an information processing system 10D of Variation 4 configured in this manner. As shown in Fig. 8, the information processing system 10D has a configuration in which an edge system 100D and a server 200 are connected via a network 500.

[0095] In this modification, the edge system 100D further includes an instruction unit 102D1, and the function of the generation unit 102D2 differs from that of the above embodiment. The other configurations and functions are the same as those in FIG. 1, which is a block diagram of the information processing system 10 according to the first embodiment, so the same reference numerals are used and the description thereof will be omitted here.

[0096] The instruction unit 102D1 determines whether the generation condition is satisfied, and if so, instructs the generation unit 102D2 to generate reference information. The generation unit 102D2 differs from the generation unit 102 of the above embodiment in that it generates reference information in response to an instruction from the instruction unit 102D1.

[0097] Fig. 9 is a sequence diagram showing an example of information processing in Modification 4. Fig. 9 differs from Fig. 3 showing the information processing of the first embodiment in that step S104D is added before step S104.

[0098] Instructing unit 102D1 determines whether the generation condition is satisfied, and if so, instructs generating unit 102D2 to generate reference information (step S104D). Generating unit 102D2 generates reference information upon receiving the instruction from instructing unit 102D1 (step S104). The processes from step S101 to step S103 and step S105 onwards are the same as those in FIG.

[0099] Fig. 10 is a flowchart showing an example of the reference information transmission process in Modification 4. Fig. 10 differs from Fig. 4 showing the reference information transmission process in the first embodiment in that step S201D is added before step S202.

[0100] Instructing unit 102D1 determines whether the conditions for generating reference information are satisfied (step S201). If the conditions for generating reference information are not satisfied (step S201: No), the process returns to step S201, and the process is repeated until the conditions are satisfied.

[0101] If the conditions for generating reference information are met (step S201: Yes), instruction unit 102D1 instructs generation unit 102D2 to generate reference information (step S201D). The processes from step S202 onwards are the same as those in FIG.

[0102] (Second embodiment) The information processing system according to the second embodiment further includes a function for collecting stream data different from video stream data. The information processing system according to the second embodiment generates reference information based on the timing at which the added stream data is acquired (generated, updated).

[0103] 11 is a block diagram showing an example of the configuration of an information processing system 10-2 according to the second embodiment. As shown in FIG. 11, the information processing system 10-2 has a configuration in which an edge system 100-2 and a server 200-2 are connected via a network 500.

[0104] The second embodiment differs from the above embodiments in that a sensor 302-2 is further connected to the edge system 100-2, that a sensed data transmission unit 104-2 (an example of a second data transmission unit) is added to the edge system 100-2, and that a sensed data reception unit 205-2 and a sensed data storage unit 223-2 are added to the server 200-2. The other configurations and functions are the same as those in FIG. 1, which is a block diagram of the information processing system 10 according to the first embodiment, and therefore the same reference numerals are used and description thereof will be omitted here.

[0105] The sensor 302-2 is a sensor that acquires detection data different from that of the camera 301. The sensor 302-2 may be any device, but may be the above-mentioned types of sensors (such as a temperature sensor or a humidity sensor), or may be a camera different from the camera 301.

[0106] In this embodiment, the sensor 302-2 acquires sensing data (an example of a second data stream) every time a certain period of time elapses or in response to a control instruction.

[0107] The sensing data transmitting unit 104-2 transmits the acquired sensing data to the server 200-2. For example, when the sensing data transmitting unit 104-2 receives sensing data from the sensor 302-2, it transmits a time information request to the generating unit 102-2 to represent time information indicating the timing of reception using time information TB, which has a different generation source from the time information TA of the video stream data. If the time information TB is obtained from a clock inside the device, the generating unit 102-2 uses the same clock to acquire time information at the time of receiving the time information request, and returns the acquired time information to the sensing data transmitting unit 104-2 as a response to the time information request. The sensing data transmitting unit 104-2 transmits stream data (time-series data) including the sensing data and time information to the server 200-2 (sensing data receiving unit 205-2).

[0108] The generation unit 102-2 generates reference information using the timing at which the time information request is received from the sensed data transmission unit 104-2 as a generation condition. The timing at which the time information request is received from the sensed data transmission unit 104-2 can be rephrased as the timing at which the sensor 302-2 acquires the sensed data.

[0109] As described above, the sensor 302-2 acquires the detection data every time a certain period of time elapses (for example, every second) or when a control instruction is received. Therefore, the generation unit 102-2 can generate the reference information using the elapse of a certain period of time or the reception of a control instruction as a generation condition, similar to the generation unit 102 of the first embodiment.

[0110] 5 shows an example in which one sensor 302-2 is provided, but multiple sensors 302-2 may be provided. In this case, the generation unit 102-2 generates the reference information using, as a generation condition, the timing at which the generation unit 102-2 receives a time information request from a detection data transmission unit 104-2 corresponding to any one of the multiple sensors 302-2.

[0111] The reference information in this embodiment is information that associates time information TB (e.g., absolute time) when registering detection data as time-series data with time information TA (e.g., frame number) of video stream data at the same timing as this time information. Such reference information makes it possible to associate time-series data (detection data) with video stream data in a chronological order. This makes it possible to more efficiently identify video stream data that corresponds to specified detection data (e.g., detection data that has undergone fluctuations).

[0112] The sensing data receiving unit 205-2 of the server 200-2 receives the sensing data transmitted from the edge system 100-2 (sensing data transmitting unit 104-2). The sensing data receiving unit 205-2 may perform preprocessing such as converting the sensing data into a format suitable for storage in the sensing data storage unit 223-2. The sensing data storage unit 223-2 stores the received sensing data.

[0113] Next, the reference information transmission process of the second embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the reference information transmission process of the second embodiment.

[0114] The generation unit 102-2 determines whether or not the sensor 302-2 has acquired the sensing data (step S301). For example, when the generation unit 102-2 receives a time information request from the sensing data transmission unit 104-2, the generation unit 102-2 determines that the sensing data has been acquired. If the sensing data has not been acquired (step S301: No), the process is repeated until the sensing data has been acquired. If the sensing data has been acquired (step S301: Yes), the generation unit 102-2 acquires the time information TA and the time information TB (step S302).

[0115] Steps S303 and S304 are the same as steps S203 to S204 (FIG. 4) in the information processing system 10 according to the first embodiment, and therefore a description thereof will be omitted.

[0116] In this way, in the second embodiment, it is possible to generate reference information that associates time information of a plurality of stream data (video stream data, data detected by a sensor, etc.), thereby making it possible to more efficiently identify one of the stream data that corresponds to the other stream data at a desired timing.

[0117] For example, when two stream data, video stream data and detection data, are obtained, the data can be identified using the following two patterns. (Pattern P1) Video stream data is provided to the user: For example, if the user checks the detection data and finds an abnormal value, the user requests video stream data with the same timing as the detection data that turned out to be abnormal. The identification unit 204 refers to the reference information and identifies the video stream data corresponding to that timing from the video storage unit 221. The request processing unit 203 provides the identified video stream data to the user. (Pattern P2) Providing detection data to the user: For example, when a user is watching a video and notices something interesting in the video at a specific timing, the user requests detection data from the same timing. The identification unit 204 refers to the reference information and identifies the detection data corresponding to that timing from the detection data storage unit 223-2. The request processing unit 203 provides the identified detection data to the user. Pattern P2 corresponds to, for example, a case where a user is watching a video and notices that smoke is rising, and requests the relevant detection data for that timing.

[0118] (Modification 1 of the second embodiment) The time information used in the reference information may have overlapping values. FIG. 13 is a diagram illustrating an example of the data structure of the reference information in Modification 1 of the second embodiment. For example, the camera 301 transmits video at 2000 frames per minute, the sensor 302-2 senses and transmits the detected data once per minute, and the generation unit 102-2 generates the reference information every 30 seconds. In this case, as shown in FIG. 13, there may be two pieces of reference information whose time information TB has a value of "20211201 12:00:00." This is because the generation unit 102-2 generates the reference information at the timing of "20211201 12:00:30," but the sensor 302-2 is not sensing at that time, and therefore the value of the time information TB is "20211201 12:00:00."

[0119] When reference information with overlapping time information is stored in the reference information storage unit 222 and the user specifies a time corresponding to the overlapping time information as the time information TC, the identification unit 204 performs, for example, the following identification process. Note that the following identification process is an example, and any other process may be used. (M1) Among the overlapping time information (for example, time information TB), only one identification result corresponding to the oldest time (previous time) of the other time information (for example, time information TA) is returned. (M2) Among the overlapping time information (for example, time information TB), only one identification result corresponding to the latest value of other time information (for example, time information TA) is returned. (M3) Return multiple identification results corresponding to all of the overlapping time information (for example, time information TB).

[0120] (Modification 2 of the second embodiment) Fig. 14 is a diagram showing an example of the data structure of the reference information in Modification 2 of the second embodiment. As shown in Fig. 14, the reference information includes identification information, time information TA1, time information TA2, and time information TB. The time information TA1 represents time information corresponding to the video stream data obtained from the camera 301. The time information TA2 represents time information when the sensor 302-2 obtained the detection data. In this case, the time information TA2 is a value obtained from an internal or inherent clock of the sensor 302-2. The time information TB represents the clock time of the edge system.

[0121] FIG. 14 shows an example of reference information when, for example, camera 301 transmits video at 2000 frames per minute, sensor 302-2 senses and transmits detected data once per minute, and generation unit 102-2 generates reference information every 30 seconds.

[0122] As shown in FIG. 14, because the generation unit 102-2 generates reference information every 30 seconds, the time information TB stores values ​​that advance by 30 seconds. At this time, the time information TA1 stores values ​​that advance by 1000 (since there are 2000 frames per minute). On the other hand, the time information TA2 has not been updated at the time the value of the time information TB becomes "20211201 12:00:30," so it has the value "20211201 12:00:00" (the time of the previous update). In this way, if the time information of the camera 301 and the sensor 302-2 has not been updated at the time the reference information is generated, the value of the immediately preceding time information may be used redundantly as the reference information.

[0123] (Third embodiment) In the third embodiment, the edge system is physically configured by multiple hardware (devices). Note that, although the following describes an example in which the edge system of the second embodiment is configured by multiple hardware, the same procedure can be applied to the case in which the edge system of the first embodiment is configured by multiple hardware.

[0124] 15 is a block diagram showing an example of the configuration of an information processing system 10-3 according to the third embodiment. As shown in FIG. 15, the information processing system 10-3 has an edge system 100-3 and a server 200-2 connected via a network 500.

[0125] In the third embodiment, an edge system 100-3 is configured by three devices: a gateway 151-3 (an example of a second device), a video device 152-3 (an example of a first device), and a sensor device 153-3. Other configurations and functions are similar to those in FIG. 1, which is a block diagram of the information processing system 10 according to the first embodiment, and therefore are denoted by the same reference numerals and will not be described here.

[0126] The gateway 151-3 includes a generation unit 102-2, a reference information transmission unit 103, and a storage unit 121. The video device 152-3 includes a video transmission unit 101. The sensor device 153-3 includes a detection data transmission unit 104-2. In this manner, in this embodiment, the units included in the edge system 100-2 of the second embodiment are configured to be distributed among multiple devices.

[0127] The gateway 151-3 has a function of aggregating reference information from a plurality of devices (video device 152-3, sensor device 153-3) that acquire stream data.

[0128] The gateway 151-3, the video device 152-3, and the sensor device 153-3 are connected to each other, for example, via a network 500. The transmission delay involved in communication between the gateway 151-3, the video device 152-3, and the sensor device 153-3 is a delay time that is sufficiently small and negligible compared to the sampling periods of the video device 152-3 and the sensor device 153-3.

[0129] The video device 152-3 and the camera 301 are connected via a control bus etc. The sensor device 153-3 and the sensor 302-2 are connected via a control bus etc.

[0130] Video transmitting unit 101 of video device 152-3 receives video stream data from camera 301 and transmits it to server 200-2. Note that video device 152-3 does not need to be directly connected to server 200-2 via network 500 or the like. In this case, video transmitting unit 101 transmits the video stream data to server 200-2 (video receiving unit 201) via gateway 151-3.

[0131] The sensed data transmitting unit 104-2 of the sensor device 153-3 receives sensed data from the sensor 302-2 and transmits it to the server 200-2 as time-series data together with time information. Upon receiving the sensed data, the sensed data transmitting unit 104-2 transmits a time information request to the generating unit 102-2 and acquires the time information TB. The sensed data transmitting unit 104-2 may acquire the time information TB from a clock or the like inside the sensor device 153-3.

[0132] The sensor device 153-3 does not have to be directly connected to the server 200-2 via the network 500. In this case, the sensed data transmitting unit 104-2 transmits the time-series data to the server 200-2 (sensed data receiving unit 205-2) via the gateway 151-3.

[0133] Note that the method of distributing each unit of the edge system 100-3 among multiple devices is not limited to the method shown in Fig. 15. For example, the units may be distributed so that two of the gateway 151-3, the video device 152-3, and the sensor device 153-3 are physically implemented by one device.

[0134] In this way, in the third embodiment, the same procedures as those in the above embodiments can be applied even when the edge system is physically configured with multiple devices.

[0135] As described above, according to the first to third embodiments, it is possible to more efficiently identify data at a desired timing.

[0136] Next, the hardware configuration of the devices (edge ​​system, server, gateway, video device, sensor device) according to the first to third embodiments will be described with reference to Fig. 16. Fig. 16 is an explanatory diagram showing an example of the hardware configuration of the devices according to the first to third embodiments.

[0137] The devices according to the first to third embodiments include a control device such as a CPU 51, a storage device such as a ROM (Read Only Memory) 52 or a RAM 53, a communication I / F 54 that connects to a network and communicates, and a bus 61 that connects each part.

[0138] The programs executed by the devices according to the first to third embodiments are provided in advance in the ROM 52 or the like.

[0139] The programs executed by the devices according to the first to third embodiments may be configured to be provided as a computer program product by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0140] Furthermore, the programs executed by the devices according to the first to third embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by the devices according to the first to third embodiments may be provided or distributed via a network such as the Internet.

[0141] The programs executed by the devices according to the first to third embodiments can cause a computer to function as each part of the server device described above. In this computer, the CPU 51 can read the programs from a computer-readable storage medium onto a main storage device and execute the programs.

[0142] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0143] 10, 10B, 10C, 10D, 10-2, 10-3 Information Processing Systems 100, 100B, 100C, 100D, 100-2, 100-3 Edge System 101, 101B, 101C Video transmission unit 102, 102D2, 102-2 generation section 102D1 Instruction section 103 Reference information transmission unit 104-2 Detection data transmission unit 121 Storage section 151-3 Gateway 152-3 Video Devices 153-3 Sensor Devices 200, 200B, 200-2 servers 201, 201B Video receiving unit 202 Reference Information Receiving Unit 203 Request Processing Unit 204 Specific section 205-2 Detection data receiver 221, 221B Video storage unit 222 Standard information storage unit 223-2 Detection data storage unit 301, 301B camera 302-2 Sensor 400 User Terminals 500 Network

Claims

1. An information processing system including a server device and one or more edge systems, The edge system includes: a first data transmitting unit that transmits a plurality of first stream data associated with the first time information to the server device; a generating unit that generates reference information that associates second time information, the second time information having a generation source different from that of the first time information, with the first time information corresponding to the first stream data selected as a reference from among the plurality of first stream data; a reference information transmitting unit that transmits the reference information to the server device; Equipped with The server device a data receiving unit that receives the plurality of first stream data from the edge system and stores the first stream data in a data storage unit; a reference information receiving unit that receives the reference information from the edge system and stores the reference information in a reference information storage unit; a request processing unit that receives a request to acquire the first stream data corresponding to the third time information, the request processing unit including third time information expressed in the same format as the second time information; an identifying unit that identifies the first stream data corresponding to the third time information included in the acquisition request based on the reference information; Equipped with Information processing system.

2. the generating unit generates the reference information every time a certain period of time elapses or when a control instruction is received. The information processing system according to claim 1 .

3. a second data transmission unit that transmits to the server device a plurality of second stream data acquired every time a certain period of time elapses or when a control instruction is received; the generating unit generates the reference information when new second stream data is acquired. The information processing system according to claim 1 .

4. the identifying unit determines the second time information that matches the third time information, or the first time information associated with the second time information whose difference from the third time information is smaller than that of other second time information, and identifies the first stream data that corresponds to the determined first time information. The information processing system according to claim 1 .

5. the identifying unit determines the first time information associated with the second time information whose difference from the third time information is smaller than that of other second time information, and identifies the first stream data corresponding to a time shifted from the determined first time information by an amount corresponding to the difference. The information processing system according to claim 1 .

6. the request processing unit transmits information indicating the identified first stream data to a source of the acquisition request. The information processing system according to claim 1 .

7. the first stream data is video stream data, the first time information is information for identifying a frame of the video stream data, the second time information and the third time information are information expressed based on a specified time; The information processing system according to claim 1 .

8. The edge system includes: a plurality of first data transmission units configured to transmit to the server device each of the plurality of first stream data items that are detected by different detection devices and associated with different pieces of first time information; the second time information has a different generation source from the first time information associated with the plurality of first stream data detected by the different detection devices; The information processing system according to claim 1 .

9. the first data transmission unit combines a plurality of first stream data detected by different detection devices and associated with different first time information, and transmits the combined first stream data to the server device; the second time information has a different generation source from the first time information associated with the plurality of first stream data detected by the different detection devices; The information processing system according to claim 1 .

10. The edge system includes: a first device including the first data transmission unit; a second device including the generation unit and the reference information transmission unit; The information processing system according to claim 1 .

11. The first time information and the second time information are expressed in different formats. The information processing system according to claim 1 .

12. a data receiving unit that receives a plurality of first stream data associated with the first time information from the edge system and stores the first stream data in a data storage unit; a reference information receiving unit that receives, from the edge system, reference information that associates second time information, which has a generation source different from that of the first time information, with the first time information corresponding to the first stream data selected as a reference from the plurality of first stream data, and stores the received reference information in a reference information storage unit; a request processing unit that receives a request to acquire the first stream data corresponding to the third time information, the request processing unit including third time information expressed in the same format as the second time information; an identification unit that identifies the first stream data corresponding to the third time information included in the acquisition request based on the reference information, A server device comprising:

13. On the computer, a data receiving step of receiving a plurality of first stream data associated with first time information from the edge system and storing the first stream data in a data storage unit; a reference information receiving step of receiving, from the edge system, reference information associating second time information, which has a generation source different from that of the first time information, with the first time information corresponding to the first stream data selected as a reference from the plurality of first stream data, and storing the reference information in a reference information storage unit; a request processing step of receiving a request to acquire the first stream data corresponding to the third time information, the request including third time information expressed in the same format as the second time information; a specifying step of specifying the first stream data corresponding to the third time information included in the acquisition request based on the reference information; A program to execute.

Images