Data processing device and method, and data processing system

Abstract

The present technology relates to a data processing apparatus and method, and a data processing system that make it possible to secure a network resource dynamically. The data processing system is provided with a client device and a resource manager. The client device transfers image data generated by an image sensor to a network. The resource manager, at the timing of generation of I frame by the image sensor, secures, on the network, a resource necessary for an object recognizing process for the image data. The present technology may be applied to a data processing apparatus and the like for controlling an application implemented on a network.

Description

[Technical Field]

[0001] The present technology relates to a data processing device and method, and a data processing system, and more particularly to a data processing device and method, and a data processing system that are capable of dynamically reserving network resources. [Background technology]

[0002] The present applicant has previously proposed an object detection system that combines a synchronous image sensor and an asynchronous DVS in Patent Document 1. A synchronous image sensor is a sensor that captures images in synchronization with a vertical synchronization signal and outputs one frame (screen) of image data at the cycle of the vertical synchronization signal. DVS is an abbreviation for Dynamic Vision Sensor, and is a sensor that treats a change in pixel luminance as an event and outputs event data that indicates the occurrence of an event. Because the DVS outputs event data when an event occurs, regardless of the vertical synchronization signal, it can be said to be an asynchronous or address-controlled image sensor.

[0003] The event data generated by DVS has extremely fine time granularity. If the large amount of data generated by DVS is indiscriminately injected into the network, the network may collapse if there are limitations on the network's capacity, and truly necessary data may not be processed correctly. On the other hand, if excessive computing resources are always reserved within the network in preparation for large amounts of data, excessive costs will be incurred. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 195769 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, IoT devices and the sensing data acquired from them have been increasingly utilized using AI (artificial intelligence). Regardless of whether DVS is used or not, resource exhaustion and wasteful energy consumption can become major problems, so a method is required to dynamically secure network resources as needed.

[0006] The present technology has been made in view of such circumstances, and makes it possible to dynamically secure network resources. [Means for solving the problem]

[0007] A data processing device according to a first aspect of the present technology includes a management module that secures resources on a network required for object recognition processing of image data transferred from an image sensor in accordance with the timing of I frame generation by the image sensor.

[0008] A data processing method according to a first aspect of the present technology includes a data processing device that reserves, on a network, resources required for object recognition processing of image data transferred from an image sensor in accordance with timing of I frame generation by the image sensor.

[0009] In the first aspect of the present technology, resources required for object recognition processing of image data transferred from an image sensor are secured on a network in accordance with the timing of I frame generation by the image sensor.

[0010] A data processing system according to a second aspect of the present technology includes a client device that transfers image data generated by an image sensor to a network, and a management module that secures resources on the network necessary for object recognition processing of the image data in accordance with the timing of I-frame generation by the image sensor.

[0011] In a second aspect of the present technology, image data generated by an image sensor is transferred to a network, and resources necessary for object recognition processing of the image data are secured on the network in accordance with the timing of I-frame generation by the image sensor.

[0012] A network includes at least two devices and a mechanism that allows information to be transmitted from one device to another. The network includes application resources for computation and storage, and network / transport resources for data transfer. Devices communicating over the network can be independent devices or internal blocks of a single device.

[0013] The management module can be realized by causing a computer to execute a program. The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0014] The data processing device may be an independent device or an internal block constituting one device. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an example of the configuration of a data processing system according to an embodiment of the present technology; [Figure 2] FIG. 1 is a diagram illustrating a typical processing flow of object recognition processing. [Figure 3] 10A and 10B are diagrams illustrating the generation of image data and event data by DVS and FBS, respectively. [Figure 4] FIG. 1 is a diagram illustrating object detection using DVS. [Figure 5] FIG. 10 is a data processing flow diagram when DVS is not implemented. [Figure 6] This is a data processing flow diagram when DVS is implemented. [Figure 7]FIG. 7 is a processing flow diagram comparing the object recognition process with DVS implementation in FIG. 5 and the object recognition process without DVS implementation in FIG. 6. [Figure 8] FIG. 10 is a block diagram illustrating an example configuration of a data processing system according to another embodiment of the present technology. [Figure 9] FIG. 2 is a block diagram illustrating a detailed configuration example of a user device. [Figure 10] FIG. 2 is a block diagram illustrating a detailed configuration example of a resource manager. [Figure 11] FIG. 10 is a block diagram illustrating a detailed configuration example of ObjDetectionEAS. [Figure 12] This is a data processing flow diagram when DVS is implemented. [Figure 13] FIG. 10 is a data processing flow diagram when DVS is not implemented. [Figure 14] FIG. 2 illustrates an example of the configuration of resources managed by a resource manager. [Figure 15] FIG. 10 is a diagram illustrating the life cycle management of application resources when DVS is not implemented. [Figure 16] FIG. 10 is a diagram illustrating the life cycle management of application resources when DVS is implemented. [Figure 17] FIG. 1 is a diagram illustrating an example of the network / transport configuration of a 5G network. [Figure 18] A diagram illustrating lifecycle management of network / transport resources in a 5G network. [Figure 19] A diagram showing an example of the network / transport configuration of the IOWN network. [Figure 20] A diagram illustrating lifecycle management of network / transport resources in an IOWN network. [Figure 21] FIG. 10 is a block diagram showing an example of the configuration of a data processing system to which an application that performs tracking processing is added. [Figure 22]FIG. 10 is a data processing flow diagram including resource adjustment / optimization of ObjTrackingEAS, which performs tracking processing. [Figure 23] FIG. 10 is a diagram illustrating an example of the structure of a workflow description. [Figure 24] 1 is a block diagram illustrating an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, with reference to the accompanying drawings, a description will be given of an embodiment of the present technology. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order. 1. Example of data processing system configuration 2. Data processing flow 3. Configuration based on edge application architecture 4. Data processing flow 5. Resource management using resource managers 6. Example of additional configuration of tracking processing module 7. Summary 8.Other Use Case Examples 9. Computer configuration example

[0017] <1. Example of data processing system configuration> FIG. 1 is a block diagram showing an example of the configuration of a data processing system according to an embodiment of the present technology.

[0018] The data processing system 1 in FIG. 1 is composed of a client device 11, a resource manager 21, an object recognition application 22, and a cloud sensor application 23, which are arranged in a network 12.

[0019] The data processing system 1 is a system that transfers data of an image captured by a client device 11 (image data) to a network 12 and executes, within the network 12, an object recognition process that recognizes an object in the image.

[0020] The network 12 includes a plurality of data processing devices and a predetermined communication network that interconnects them. Examples of the predetermined communication network include a local area network (LAN), a wide area network (WAN), the Internet, mobile communication networks such as so-called 4G lines and 5G lines, and the all-photonics network proposed by IOWN Global Forum, Inc. The data processing device is composed of, for example, a sensor device, a router, a modem, a hub, a bridge, a switching hub, a base station control device, an exchange, a server device, and the like, and has a network connection function for connecting to the predetermined communication network and a data processing function for processing data acquired via the network.

[0021] The resource manager 21, the object recognition application 22, and the cloud sensor application 23 are applications (modules) that run on predetermined data processing devices within the network 12. The network 12 includes an edge environment (edge-side cloud) close to the client device 11 that injects image data into the network 12, and a cloud on the core network side (center-side cloud). The object recognition application 22 is typically run in an edge environment because it must execute object recognition processing while satisfying predetermined latency requirements.

[0022] The client device 11 is configured to include a DVS 31 and a frame-based sensor 32. However, the DVS 31 is not essential and may be omitted. Below, two cases will be described: when the client device 11 is equipped with the DVS 31, and when the client device 11 is not equipped with the DVS 31. The imaging ranges of the DVS 31 and the frame-based sensor 32 are adjusted to be the same.

[0023] The DVS31 is an event sensor that has pixels that photoelectrically convert optical signals to output pixel signals, and outputs temporal brightness changes in the optical signals as event data based on the pixel signals. A typical image sensor captures images in synchronization with a vertical synchronization signal and outputs image data, which is image data in frame (screen) units, at the frequency of that vertical synchronization signal. However, the DVS31 outputs event data only when an event occurs, so it can be said to be an asynchronous or address-controlled image sensor. The DVS is also known as an event-based sensor.

[0024] In the DVS 31, for example, a voltage signal corresponding to the logarithmic value of the amount of light incident on each pixel is detected as a pixel signal. When the logarithmic luminance change value represented by the pixel signal exceeds a predetermined threshold c and changes to brighter, the DVS 31 outputs “+1” representing a positive luminance change, and when the logarithmic luminance change value represented by the pixel signal exceeds the predetermined threshold c and changes to darker, the DVS 31 outputs “−1” representing a negative luminance change.

[0025] Event data is represented, for example, in the following format called AER (Address-Event Representation) format. e = (x, y, p, t) ·········(1)

[0026] In equation (1), x and y represent the coordinates of the pixel where a luminance change occurred. The time t of an event is a timestamp that indicates the time when the event occurred, and is expressed, for example, as the count value of a counter based on a predetermined clock signal in the sensor. The timestamp corresponding to the timing when an event occurred can be said to be time information that indicates the (relative) time when the event occurred, as long as the interval between events is maintained as it was when the events occurred.

[0027] The polarity p represents the direction of the luminance change when a luminance change (change in light intensity) exceeding a predetermined threshold c occurs as an event, and indicates whether the luminance change is a positive change (hereinafter also referred to as positive) or a negative change (hereinafter also referred to as negative). The polarity p of an event is represented, for example, by "+1" when it is positive and by "-1" when it is negative.

[0028] As described above, the DVS31 outputs only the position coordinates, polarity, and time information of pixels where a change in brightness has been detected. Because the DVS31 generates and outputs only the net change (difference) in the form of position coordinates, polarity, and time information, there is no redundancy in the amount of data and it has high time resolution on the order of microseconds. Furthermore, because the amount of information is small, it consumes less power than an image sensor that outputs image data in frames, and when processing the data, there is no unnecessary processing load and processing time can be shortened. Because it is capable of high-speed, low-latency data output, it is possible to obtain the exact time when an event occurred.

[0029] The frame-based sensor 32 corresponds to the general image sensor described above, and is a sensor that captures images in synchronization with a vertical synchronization signal and generates image data for moving images frame by frame at the period of the vertical synchronization signal. The frame-based sensor 32 can be of any type as long as it generates image data frame by frame, and can be configured, for example, as an image sensor that receives RGB light and generates an RGB image, or an image sensor that receives IR light and generates an IR image. For simplicity, the frame-based sensor 32 will be referred to as FBS 32 in the following description.

[0030] The FBS 32 encodes (compresses and encodes) the image data of the captured moving image using the LongGOP compression method, which compresses the image data across multiple frames. Encoders that use the LongGOP compression method include, for example, MPEG-4 AVC and H.264. The encoded image stream, which is the encoded moving image stream, is transferred from the client device 11 to the object recognition application 22.

[0031] The FBS 32 detects the generation of an I frame among the I frames, P frames, and B frames that make up a GOP (Group Of Picture), and transmits an I frame transfer timing notification indicating the timing at which the I frame is to be transferred to the resource manager 21. Although it is possible to forcibly generate an I frame at a fixed interval, in this embodiment, the encoder of the FBS 32 generates an I frame when a scene change is detected in the video based on a scene change detection parameter. Therefore, the FBS 32 transmits an I frame transfer timing notification to the resource manager 21 when an I frame is generated due to a scene change in the video.

[0032] When the DVS 31 is implemented in the client device 11, the DVS 31 can observe brightness changes at a time granularity that cannot be detected with frame-based image data as described above, and therefore can detect the entry of an object into the imaging range corresponding to a scene change earlier than image data from the FBS 32. When the DVS 31 detects the entry of a new object into the imaging range based on the event data, it transmits object candidate area determination information to the resource manager 21. The object candidate area determination information includes the number of new candidate areas required in response to the detected new object and position information identifying each candidate area.

[0033] When the resource manager 21 receives an I frame transfer timing notification or object candidate area determination information from the client device 11, it reserves (secures) resources for the object recognition application 22 within the network 12 and executes the object recognition application 22. After securing the resources for the object recognition application 22, if the resource manager 21 receives a request from the cloud sensor application 23 to release the secured resources, it releases the resources for the object recognition application 22.

[0034] The object recognition application 22 performs transport processing, decoding processing, object detection, and object recognition processing of the encoded image stream. The transport processing, decoding processing, object detection, and object recognition processing are configured as separate processing modules and can be started and executed independently. The transport processing module acquires the encoded image stream from the FBS 32. The decoding processing module decodes the encoded image stream. The object detection module detects objects in the decoded video. The object recognition processing module classifies the detected objects. Each module may be a separate application. The object recognition application 22 notifies the cloud sensor application 23 of the recognition results of the object recognition processing.

[0035] The cloud sensor application 23 performs a predetermined application process based on the recognition result of the object recognition process on the moving image performed by the object recognition application 22. After performing the predetermined application process using the recognition result, the cloud sensor application 23 notifies the resource manager 21 of a resource release request.

[0036] As described above, the data processing system 1 is configured to execute the object recognition process for recognizing objects in moving images captured by the FBS 32 using the object recognition application 22 on the network 12.

[0037] Object recognition processing is a high-load process, and unnecessary processing must be reduced as much as possible. A system that constantly sends image data from FBS32 as baseband or intra-encoded data and constantly runs object recognition processing without knowing whether new objects have been added will waste resources. If the encoder installed in FBS32 forcibly generates I frames at regular intervals and runs object recognition processing in conjunction with I frame generation, the load will be slightly reduced compared to running object recognition processing constantly, but there will still be a lot of wasted resources.

[0038] Latency requirements for object recognition processing running in edge environments are expected to become increasingly strict in the future. While it would be possible to constantly reserve sufficient resources in excess of what is needed to eliminate the delays associated with advance preparations for reserving resources, constantly reserving excessive resources can lead to major problems such as resource depletion and energy consumption. Therefore, a method is needed that can dynamically reserve resources as needed, as much as possible.

[0039] The data processing system 1 dynamically and sequentially secures the necessary and sufficient amount of resources, such as transport resources for transferring the encoded image stream, and computational and memory resources required to execute object recognition processing, only when a new object (object) enters the imaging range of the FBS32 and a scene change occurs.

[0040] FIG. 2 is a diagram illustrating a typical processing flow of object recognition processing.

[0041] Object recognition is the process of locating and classifying objects in an image. CNN (Convolutional Neural Network)-based methods have been proposed for object detection and classification. R-CNN (Regions with Convolutional Neural Networks), a representative method, performs the following steps in that order: candidate region determination, feature extraction, and object classification. In candidate region determination, a region proposal unit detects regions (candidate regions) within an image that may contain objects. Candidate regions are converted into fixed-size regions. In feature extraction, a CNN feature extraction unit extracts CNN features from candidate regions. In object classification, an SVM classifier classifies objects based on the extracted features. Candidate region determination corresponds to object detection, while feature extraction and object classification correspond to object recognition. Generally, feature extraction and object classification are performed in parallel for each candidate region to minimize processing delays.

[0042] As described above, object recognition processing must be performed in parallel for each candidate region. Therefore, if it is possible to determine in advance how many new objects will appear in a new I-frame image, it may be possible to accurately estimate how many processing resources will be required for the recognizer (inference engine) that must be run simultaneously.

[0043] FIG. 3 shows how the DVS 31 and FBS 32 generate image data and event data in time series, respectively.

[0044] The FBS 32 generates frame images FR1, FR2, FR3, and FR4 at times t10, t20, t30, and t40, respectively. The time intervals between times t10, t20, t30, and t40 correspond to the frame capture period.

[0045] The DVS31 generates event data when it detects a change in brightness that occurs in response to the movement of a subject, etc. The bars on the time axis represent event data, and the bars are connected when event data occurs consecutively.

[0046] The encoder of FBS32 encodes frame image FR1 at time t10 as an I frame. The encoder encodes frame image FR2 at time t20 as a P frame using motion compensated prediction from the I frame at time t10. Frame image FR3 at time t30 is encoded as a P frame using motion compensated prediction from the I frame at time t10 or the P frame at time t20. Object a appears in frame image FR1 at time t10, and object a appears to be moving in frame image FR2 at time t20 and frame image FR3 at time t30.

[0047] In addition to object a, object b and object c appear in frame image FR4 at time t40. The encoder of FBS32 detects that a scene change has occurred in frame image FR4 due to the entry of new objects b and c into the imaging range, and encodes it as an I frame. That is, the encoder stops calculations such as block matching for motion compensation prediction from the immediately preceding P frame that were started in an attempt to encode frame image FR4 as a P frame, and switches to encoding as an I frame. The timing of this detection process is always after the frame image FR4 at time t40 is captured within FBS32.

[0048] In contrast, when we look at the event data generated by the DVS31, the event data EVa1 is event data generated in response to the entry of object a included in frame image FR1 at time t10 into the imaging range. Each of the event data EVa2, EVa3, and EVa4 is event data generated in response to the movement of object a within the imaging range.

[0049] Assume that new objects b and c, different from object a, enter the imaging range at a predetermined time t34 within the frame capture period from time t30 to time t40. Frame image FR3' shows an image captured by FBS32 at time t34. At time t34, DVS31 generates event data EVb1 associated with the entry of object b into the imaging range, and generates event data EVc1 associated with the entry of object c into the imaging range.

[0050] In typical network image data transfers, an encoder detects an I-frame at time t40, encodes it as an I-frame (intra-frame compression, etc.), and transmits the image data of that I-frame to network 12. The object recognition server in network 12 receives and decodes the I-frame at time t40 and executes recognition engine processing (candidate area determination processing, object classification processing, etc.) before detecting the presence of a new object. Therefore, the object recognition server does not know how many new objects are included until the candidate area determination processing is performed, and therefore cannot estimate in advance the computational resources required for feature extraction within each candidate area, object classification processing, etc. Generally, when latency requirements are strict, simultaneous parallel processing proportional to the number of objects is required.

[0051] In response to this, if the client device 11 of the data processing system 1 does not have the DVS 31 installed, it traps the result of the I-frame generation determination based on the threshold value (scene change detection threshold value) for determining whether to generate an I-frame in the FBS 32 encoder, and sends an I-frame transfer timing notification to the resource manager 21 immediately before the encoder starts generating an I-frame. The resource manager 21 quickly reserves resources for the object recognition application 22, and prepares so that the object recognition application 22 can immediately start object recognition processing corresponding to the addition of an object. However, in this case, since it is not known in advance how many new objects will be included, it is not possible to estimate in advance the computational resources required for feature extraction for each object, object classification processing, etc.

[0052] FIG. 4 is a diagram illustrating object detection by the DVS 31 in the frame capture period from time t30 to time t40 in FIG.

[0053] Since DVS31 can capture data with fine-grained temporal resolution, it can distinguish the temporal transition of candidate regions of an object at smaller time intervals than the frame capture period of FBS32. Specifically, when DVS31 is implemented in the client device 11, DVS31 can detect candidate regions corresponding to new objects b and c that are different from the movement of object a observed at time t30 at time t34. This makes it possible to secure computational resources for object classification processing from the necessary feature extraction processing earlier than in the case of only FBS32 described above. That is, between time t30 and time t40, it is possible to estimate and secure in advance how much object recognition processing resources are required in the edge environment of the network 12, so that the resource securing process can be performed safely and accurately. Here, since an image frame is required for high-precision recognition processing that uses color in combination, object recognition processing in the edge environment is premised on being executed based on the I frame due to the scene change of FBS32. DVS31 is merely treated as "used in combination" to pre-secure the resources required for the recognition of the I frame of FBS32.

[0054] As described above, when DVS31 is implemented in the client device 11, DVS31 can detect new objects with fine temporal granularity and pre-secure the necessary resources. In other words, compared to the case of performing it at the timing of I frame generation detection, the candidate region determination process can be performed earlier and with finer temporal granularity. By performing CNN feature extraction processing and object classification processing based on the determination process of candidate regions in the client device 11 based on this event data, faster object recognition processing becomes possible compared to object recognition processing based only on I frame generation detection, and the processing resources (number) of the recognizer that must be executed simultaneously can also be accurately estimated.

[0055] <2. Data Processing Flow> <When DVS is not implemented> 5, a data processing flow for object recognition when the client device 11 does not include the DVS 31 will be described. Apart from this processing, the FBS 32 continues to capture images of the subject.

[0056] First, in step S11, the FBS 32 of the client device 11 encodes captured video using the LongGOP compression method. When a scene change is detected during encoding, the FBS 32 detects the generation of an I-frame and transmits an I-frame transfer timing notification to the resource manager 21.

[0057] In step S12, the resource manager 21 receives an I-frame transfer timing notification from the FBS 32. The resource manager 21 reserves resources necessary for transport processing, decoding processing, and object detection, and executes the transport processing module, decoding processing module, and object detection module of the object recognition application 22.

[0058] In step S13, the FBS 32 uplinks (sends) the encoded image stream to the transport processing module of the object recognition application 22.

[0059] In step S14, the object recognition application 22 executes a series of processes from receiving the encoded image stream to recognizing the object. Specifically, the transport processing module, decoding processing module, and object detection module, which were executed earlier, sequentially receive the encoded image stream, decode it, and detect objects. Then, based on the result of the object detection, the resource manager 21 is requested to secure resources required for object recognition processing and execute the module. The object recognition processing module executed in response to the request executes the object recognition processing and notifies the cloud sensor application 23 of the recognition result.

[0060] In step S15, when the resource manager 21 is notified from the object recognition application 22 of the request for securing resources necessary for object recognition processing and module execution, it secures the resources and causes the object recognition processing module to execute. The object recognition processing module is executed in parallel for the number of candidate regions for object recognition.

[0061] In step S16, the cloud sensor application 23 performs predetermined application processing based on the recognition result of the object recognition processing transmitted from the object recognition application 22. After executing the predetermined application processing, the cloud sensor application 23 notifies the resource manager 21 of a resource release request.

[0062] In step S17, the resource manager 21 receives the resource release request from the cloud sensor application 23. The resource manager 21 stops the execution of each module of the object recognition application 22 and releases the resources. The execution of the transport processing module, the decoding processing module, the object detection module, and the object recognition processing module is stopped, and their resources are released.

[0063] <When implementing DVS> Next, referring to FIG. 6, the data processing flow of object recognition when the client device 11 implements the DVS 31 will be described. Separately from this processing, event detection by the DVS 31 and imaging of the subject by the FBS 32 are continuously executed.

[0064] First, in step S31, the DVS 31 of the client device 11 detects a new object that has entered the imaging range and notifies the resource manager 21 of the candidate region determination information of the detected object.

[0065] In step S32, the resource manager 21 receives the object candidate area determination information from the DVS 31, reserves resources necessary for object recognition processing according to the number of candidate areas, and causes the object recognition application 22 to execute the object recognition processing module.

[0066] In step S33, the FBS 32 of the client device 11 encodes the captured video using the LongGOP compression method. When a scene change is detected during encoding, the FBS 32 detects the generation of an I-frame and transmits an I-frame transfer timing notification to the resource manager 21.

[0067] In step S34, the resource manager 21 receives the I frame transfer timing notification from the FBS 32. The resource manager 21 reserves resources necessary for transport processing and decoding processing, and causes the transport processing module and decoding processing module of the object recognition application 22 to execute.

[0068] In step S35, the FBS 32 uplinks (sends) the encoded image stream to the transport processing module of the object recognition application 22.

[0069] In step S36, the object recognition application 22 executes a series of processes from receiving the encoded image stream to object recognition. The resources required for the object recognition process have already been secured by the process in step S32 described above. Using the secured resources, the object recognition process, which includes feature extraction and object classification, is executed in parallel for the number of candidate regions for object recognition. The object recognition application 22 notifies the cloud sensor application 23 of the recognition results of the object recognition process.

[0070] In step S37, the cloud sensor application 23 performs a predetermined application process based on the recognition result of the object recognition process transmitted from the object recognition application 22. After executing the predetermined application process, the cloud sensor application 23 notifies the resource manager 21 of a resource release request.

[0071] In step S38, the resource manager 21 receives a resource release request from the cloud sensor application 23. The resource manager 21 stops the execution of each module of the object recognition application 22 and releases the resources. The execution of each of the transport processing module, the decoding processing module, and the object recognition processing module is stopped, and their resources are released.

[0072] Comparing the process of FIG. 5 when the DVS 31 is not implemented with the process of FIG. 6 when the DVS 31 is implemented, when the DVS 31 is implemented, the process of step S31 is added. Along with that, the process in which the resource manager 21 secures and executes the resources of each module is divided into the process of step S32 before receiving the I-frame transfer timing notification and the process of step S34 after receiving the I-frame transfer timing notification and executed.

[0073] <Comparison with and without DVS implementation> FIG. 7 is a process flow comparing the case where the DVS 31 shown in FIG. 5 is not implemented and the case where the DVS 31 shown in FIG. 6 is implemented for the candidate region determination process, the feature amount extraction process, and the object classification process, which are the specific processes of the object recognition process.

[0074] The upper part of FIG. 7 is the detailed process flow in the case where the DVS 31 shown in FIG. 5 is not implemented. The lower part of FIG. 7 is the detailed process flow in the case where the DVS 31 shown in FIG. 6 is implemented.

[0075] In the process when the DVS 31 is not implemented, the FBS 32 uplinks (transmits) the encoded image stream to the transport processing module of the object recognition application 22 in step S51.

[0076] In step S52, the object detection module of the object recognition application 22 detects a new object that has entered the imaging range, and notifies the resource manager 21 of candidate area determination information for the detected object.

[0077] In step S53, the resource manager 21 receives the candidate area determination information for the object from the object recognition application 22, and secures and executes the resources required for object recognition processing according to the number of candidate areas. As a result, resources for the feature extraction processing module and object classification processing module are secured for each candidate area, and the feature extraction processing module and object classification processing module for each candidate area are executed.

[0078] Then, in step S54, the feature extraction processing module and the object classification processing module execute the feature extraction processing and the object classification processing for each candidate region. The recognition result obtained as a result of the object classification processing is notified to the cloud sensor application 23.

[0079] On the other hand, in the process when the DVS 31 is installed, in step S71, the DVS 31 detects a new object that has entered the imaging range and notifies the resource manager 21 of candidate area determination information for the detected object.

[0080] In step S72, the resource manager 21 receives the candidate area determination information for the object from the DVS 31, and allocates and executes resources necessary for object recognition processing according to the number of candidate areas. Resources for the feature extraction processing module and object classification processing module are allocated for each candidate area, and the feature extraction processing module and object classification processing module for each candidate area are executed.

[0081] In step S73, the FBS 32 uplinks (sends) the encoded image stream to the transport processing module of the object recognition application 22.

[0082] In step S74, the feature extraction processing module and the object classification processing module execute the feature extraction processing and the object classification processing for each candidate region. The recognition result obtained as a result of the object classification processing is notified to the cloud sensor application 23.

[0083] 7, when the DVS31 is not implemented, the object recognition application 22 executes the process in step S52, whereas when the DVS31 is implemented, the DVS31 executes the process in step S71. In other words, there is no need to reserve resources on the network 12 for the object detection module that performs the candidate area determination process (or a process equivalent thereto). Furthermore, when the DVS31 is implemented, resources for the feature extraction process and object classification process for each candidate area can be reserved before receiving the encoded image stream from the FBS32, allowing for a margin in resource reservation and improving the reliability of resource reservation. Furthermore, because resources for the feature extraction process and object classification process can be reserved before receiving the encoded image stream, overall processing delay can be reduced.

[0084] <3. Configuration based on edge application architecture> <Data Processing System> FIG. 8 is a block diagram showing an example of the configuration of a data processing system according to another embodiment of the present technology.

[0085] The data processing system 100 shown in FIG. 8 is a configuration example in which the above-mentioned data processing system 1 is realized using the edge application architecture (3GPP TS 23.558 “Architecture for enabling Edge Applications (Release 17)”) that is being standardized by 3GPP (Third Generation Partnership Project)-SA6, a standardization organization for mobile communications.

[0086] The data processing system 100 includes a user device 111, a resource manager (ResourceManager) 121, an ObjDetectionEAS 122, and a cloud sensor application (CloudSensorApplication) 123, which are arranged in a network (cloud) 112.

[0087] The user device 111 includes a DVS 131, a FrameBasedSensor (FBS) 132, and an ObjDetectionEAC 133. Note that, similar to the data processing system 1 described above, the DVS 131 may be omitted.

[0088] The edge application architecture defines an EAC (EdgeAppClient) and an EAS (EdgeAppServer), and the EAS is paired with an Application Client on a user device (Use Equipment). The EAC is an application that executes the client function of a specific application on the user device, and the EAS is an application that executes the server function of that application in an Edge environment (Edge Data Network).

[0089] The ObjDetectionEAC 133 is configured by an EAC of the edge application architecture, and the ObjDetectionEAS 122 is configured by an EAS of the edge application architecture. The resource manager 121 and the cloud sensor application 123 are newly introduced entities for realizing this technology. The resource manager 121 may be an application executed in an edge environment or an application executed in the cloud.

[0090] 1, the DVS 131 of the user device 111 detects a change in brightness in the same imaging range as the FBS 132 as an event and outputs the event data to the ObjDetectionEAC 133. The FBS 132, like the FBS 32 of FIG. 1, captures moving images at a frame capture period and outputs baseband image data to the ObjDetectionEAC 133.

[0091] Based on event data supplied from the DVS 131, the ObjDetectionEAC 133 detects a new object that has entered the imaging range and notifies the resource manager 121 of candidate area determination information for the detected object. The ObjDetectionEAC 133 also encodes the video captured by the FBS 132 using the LongGOP compression method and transmits the resulting encoded image stream to the ObjDetectionEAS 122. During encoding, the ObjDetectionEAC 133 detects the generation of an I-frame accompanying a scene change and transmits an I-frame transfer timing notification to the resource manager 121.

[0092] The resource manager 121 secures resources for and executes the transport and decode processing modules of ObjDetectionEAS122 based on an I frame transfer timing notification from ObjDetectionEAC133. The resource manager 121 also secures resources for and executes the feature extraction and object classification processing modules of ObjDetectionEAS122 based on object candidate area determination information notified from either ObjDetectionEAC133 or ObjDetectionEAS122. Furthermore, when the resource manager 121 is notified of a resource release request from the cloud sensor application 123, it stops the execution of the transport and decode processing modules, feature extraction and object classification processing modules, etc. currently running in ObjDetectionEAS122 and releases the resources.

[0093] The ObjDetectionEAS 122 performs transport processing, decoding processing, and object recognition processing (feature extraction processing and object classification processing) of the encoded image stream transmitted from the ObjDetectionEAC 133. Furthermore, if the user device 111 does not have the DVS 131 implemented, the ObjDetectionEAS 122 also performs object detection based on the decoded image data, in other words, object candidate area determination processing, and notifies the resource manager 121 of the object candidate area determination information. The ObjDetectionEAS 122 notifies the cloud sensor application 123 of the recognition result of the object recognition processing.

[0094] The cloud sensor application 123 performs a predetermined application process based on the recognition result of the object recognition process on the video image performed by the ObjDetectionEAS 122. After performing the predetermined application process using the recognition result, the cloud sensor application 123 notifies the resource manager 121 of a resource release request.

[0095] <User device> FIG. 9 is a block diagram showing an example of a detailed configuration of the user device 111.

[0096] The ObjDetectionEAC 133 of the user device 111 is equipped with a DVS data processing module 151 that processes event data from the DVS 131 and an image frame encoder module 152 that processes image data from the FBS 132.

[0097] The DVS data processing module 151 analyzes the event data from the DVS 131, determines the candidate area of ​​a new object that has entered the imaging range, and notifies the resource manager 121 running in an edge environment or on the cloud of the candidate area determination information for the object.

[0098] The image frame encoder module 152 encodes the baseband image data from the FBS 132. At that time, the image frame encoder module 152 determines whether to generate an I frame using, for example, a scene change detection algorithm, and if it determines to generate an I frame, sends an I frame transfer timing notification to the resource manager 121 before encoding the I frame. The image frame encoder module 152 also transmits an encoded image stream to ObjDetectionEAS 122, a server-side device corresponding to ObjDetectionEAC 133 and operating in an edge environment. If ObjDetectionEAS 122 notifies the module 152 to stop transferring the encoded image stream, the transmission of the encoded image stream is stopped.

[0099] FIG. 10 is a block diagram showing an example of the detailed configuration of the resource manager 121. As shown in FIG.

[0100] The resource manager 121 runs in an edge environment or on the cloud, and manages resources for calculation and storage of the ObjDetectionEAS 122. A transport and decoder resource management module 171 and a feature extraction and classification processing resource management module 172 are implemented in the resource manager 121. Note that in FIG. 10, the "transport and decoder resource management module" is written as the "transport & decoder resource management module," and the "feature extraction and classification processing resource management module" is written as the "feature extraction & classification processing resource management module," with "and" written as "&." Similar notations are used in FIG. 11 and subsequent figures.

[0101] The transport and decoder resource management module 171 receives an I frame transfer timing notification from the ObjDetectionEAC 133, and secures resources for and executes the transport and decode processing module 191 (FIG. 11) of the ObjDetectionEAS 122. The resources for the transport and decode processing module 191 are secured in time for the transfer of the I frame of the encoded image stream from the ObjDetectionEAC 133. Furthermore, if the user device 111 does not have the DVS 131 implemented, the transport and decoder resource management module 171 also secures resources for and executes the candidate area determination processing module 192 (FIG. 11) of the ObjDetectionEAS 122.

[0102] The feature extraction and classification processing resource management module 172 receives object candidate region determination information. The object candidate region determination information is notified from the ObjDetectionEAC 133 if the DVS 131 is implemented in the user device 111, and is notified from the ObjDetectionEAS 122 if the DVS 131 is not implemented in the user device 111.

[0103] When the DVS 131 is implemented, the feature extraction and classification processing resource management module 172 secures resources for and executes the feature extraction and object classification processing module 193 (FIG. 11) based on the object candidate area determination information notified by the ObjDetectionEAC 133. In addition, the feature extraction and classification processing resource management module 172 transfers the notified object candidate area determination information to the ObjDetectionEAS 122.

[0104] On the other hand, if the DVS 131 is not implemented, the feature extraction and classification processing resource management module 172 secures and executes resources for the feature extraction and object classification processing module 193 (FIG. 11) based on the object candidate area determination information from the ObjDetectionEAS 122. The resources for the feature extraction and object classification processing module 193 are secured so that the feature extraction processing and object classification processing are executed in parallel for each candidate area.

[0105] When transport and decoder resource management module 171 receives a resource release request from cloud sensor application 123, it stops execution of transport and decode processing module 191 of ObjDetectionEAS 122 and releases resources. When feature extraction and classification processing resource management module 172 receives a resource release request from cloud sensor application 123, it stops execution of feature extraction and object classification processing module 193 of ObjDetectionEAS 122 and releases resources. If candidate area determination processing module 192 is also running, feature extraction and classification processing resource management module 172 also stops execution of candidate area determination processing module 192 and releases resources.

[0106] FIG. 11 is a block diagram showing a detailed configuration example of the ObjDetectionEAS 122. As shown in FIG.

[0107] The ObjDetectionEAS 122 is implemented with a transport and decoding processing module 191, a candidate region determination processing module 192, and a feature extraction and object classification processing module 193. As described above, the execution of each module of the ObjDetectionEAS 122 is started and stopped by the resource manager 121. The ObjDetectionEAS 122 operates in an edge environment.

[0108] The transport and decode processing module 191 receives and decodes the encoded image stream from the ObjDetectionEAC 133. The baseband image data obtained by decoding is supplied to the feature extraction and object classification processing module 193, and if the DVS 131 is not implemented, is also supplied to the candidate area determination processing module 192. When the transport and decode processing module 191 stops its own execution, it notifies the ObjDetectionEAC 133 of the user device 111 to stop transferring the encoded image stream.

[0109] The candidate area determination processing module 192 is started and executed only when the DVS 131 is not implemented in the user device 111, and is not started and executed when the DVS 131 is implemented.

[0110] Candidate area determination processing module 192 performs candidate area determination processing using baseband image data supplied from transport and decode processing module 191. That is, candidate area determination processing module 192 detects a new object in the captured image and notifies feature extraction and object classification processing module 193 of candidate area determination information for the detected object. The object candidate area determination information is also notified to feature extraction and classification processing resource management module 172 of resource manager 121.

[0111] When the DVS 131 is not implemented in the user device 111, the feature quantity extraction and object classification processing module 193 is supplied with the object candidate region determination information from the candidate region determination processing module 192. On the other hand, when the DVS 131 is implemented, the object candidate region determination information is supplied from the feature quantity extraction and classification processing resource management module 172 of the resource manager 121.

[0112] The feature quantity extraction and object classification processing module 193 executes the feature quantity extraction processing and the object classification processing in parallel for each candidate region based on the baseband image data from the transport and decoding processing module 191 and the object candidate region determination information from the feature quantity extraction and object classification processing module 193 or the resource manager 121. The feature quantity extraction and object classification processing module 193 notifies the cloud sensor application 123 of the result of the object classification processing, that is, the recognition result of the object recognition processing.

[0113] <4. Data Processing Flow> <When Implementing DVS> Next, referring to FIG. 12, the data processing flow of object recognition when the user device 111 implements the DVS 31 will be described. Separately from this process, the event detection by the DVS 131 and the imaging of the subject by the FBS 132 are continuously executed.

[0114] First, in step S101, the DVS data processing module 151 analyzes the event data supplied from the DVS 131 and detects an object within the imaging range. The DVS data processing module 151 determines the candidate region of the object and notifies the resource manager 121 of the object candidate region determination information.

[0115] In step S102, feature extraction and classification processing resource management module 172 receives the object candidate area determination information and, based on the candidate area determination information, allocates resources for and executes feature extraction and object classification processing module 193. Feature extraction and object classification processing module 193 is allocated and executed according to the number of candidate areas. Subsequently, in step S103, feature extraction and classification processing resource management module 172 notifies (transfers) the object candidate area determination information notified by DVS data processing module 151 to feature extraction and object classification processing module 193 of ObjDetectionEAS 122.

[0116] In step S104, the image frame encoder module 152 encodes the baseband image data supplied from the FBS 132. At this time, the image frame encoder module 152 determines whether to generate an I-frame using a scene change detection algorithm. If the image frame encoder module 152 determines to generate an I-frame, it sends an I-frame transfer timing notification to the transport and decoder resource management module 171 of the resource manager 121 before encoding the I-frame.

[0117] In step S105, the transport and decoder resource management module 171 receives the I frame transfer timing notification from the image frame encoder module 152 and reserves and executes resources for the transport and decode processing module 191 of the ObjDetectionEAS 122. The resources for the transport and decode processing module 191 are reserved before the I frame of the encoded image stream is transferred from the image frame encoder module 152 in the next step S106.

[0118] In step S106, the image frame encoder module 152 uplinks (transmits) the encoded image stream obtained by encoding the captured baseband image data to the transport and decode processing module 191 of the ObjDetectionEAS 122.

[0119] In step S107, the transport and decoding processing module 191 performs transport processing and decoding of the encoded image stream. This receives the encoded image stream transmitted from the image frame encoder module 152, decodes it, and converts it into baseband image data. The baseband image data is supplied to the feature extraction and object classification processing module 193.

[0120] In step S108, feature extraction and object classification processing module 193 executes feature extraction processing and object classification processing in parallel for each candidate region based on the baseband image data from transport and decode processing module 191 and the object candidate region determination information from feature extraction and classification processing resource management module 172. Feature extraction and object classification processing module 193 notifies cloud sensor application 123 of the object classification processing result, i.e., the object recognition result.

[0121] In step S109, the cloud sensor application 123 performs a predetermined application process based on the recognition result of the object recognition process from the feature extraction and object classification processing module 193. After executing the predetermined application process, the cloud sensor application 123 notifies the resource manager 121 of a resource release request.

[0122] In step S110, the resource manager 121 receives a resource release request from the cloud sensor application 123 and releases the resources of the transport and decoding processing module 191 and the feature extraction and object classification processing module 193 of the ObjDetectionEAS 122. More specifically, the transport and decoder resource management module 171 stops the execution of the transport and decoding processing module 191 and releases the resources. The feature extraction and classification processing resource management module 172 stops the execution of the feature extraction and object classification processing module 193 and releases the resources.

[0123] In step S111, before stopping its own execution, the transport and decoding processing module 191 notifies the image frame encoder module 152 to stop the transfer of the encoded image stream.

[0124] When the user device 111 implements the DVS 31, the data processing of object recognition is executed as described above.

[0125] <When the DVS is not implemented> Next, referring to FIG. 13, the data processing flow of object recognition when the user device 111 does not implement the DVS 31 will be described. Separately from this processing, the imaging of the subject by the FBS 132 is continuously executed.

[0126] First, in step S131, the image frame encoder module 152 encodes the baseband image data supplied from the FBS 132. At that time, the image frame encoder module 152 performs determination of the generation of an I frame by a scene change detection algorithm. When the image frame encoder module 152 determines to generate an I frame, before encoding the I frame, it transmits an I frame transfer timing notification to the transport and decoder resource management module 171 of the resource manager 121.

[0127] In step S132, transport and decoder resource management module 171 receives the I frame transfer timing notification from image frame encoder module 152, and secures and executes resources for transport and decode processing module 191 of ObjDetectionEAS 122. The resources for transport and decode processing module 191 are secured before an I frame of the encoded image stream is transferred from image frame encoder module 152 in step S134, which will be described later.

[0128] In step S133, the transport and decoder resource management module 171 secures resources for and executes the candidate area determination processing module 192.

[0129] In step S134, the image frame encoder module 152 uplinks (transmits) the encoded image stream obtained by encoding the captured baseband image data to the transport and decode processing module 191 of the ObjDetectionEAS122.

[0130] In step S135, the transport and decoding processing module 191 transports and decodes the encoded image stream. This receives and decodes the encoded image stream transmitted from the image frame encoder module 152 and converts it into baseband image data. The baseband image data is supplied to the candidate area determination processing module 192 and the feature extraction and object classification processing module 193.

[0131] In step S136, candidate area determination processing module 192 performs candidate area determination processing using the baseband image data supplied from transport and decode processing module 191. That is, candidate area determination processing module 192 detects an object in the captured image and notifies feature extraction and object classification processing module 193 of candidate area determination information for the detected object. The object candidate area determination information is also notified to feature extraction and classification processing resource management module 172 of resource manager 121.

[0132] In step S137, the feature extraction and classification processing resource management module 172 secures resources for and executes the feature extraction and object classification processing module 193 based on the object candidate area determination information from the candidate area determination processing module 192.

[0133] In step S138, feature extraction and object classification processing module 193 executes feature extraction processing and object classification processing in parallel for each candidate area based on the baseband image data from transport and decode processing module 191 and the object candidate area determination information from candidate area determination processing module 192. Feature extraction and object classification processing module 193 notifies cloud sensor application 123 of the object classification processing result, i.e., the object recognition result.

[0134] In step S139, the cloud sensor application 123 performs a predetermined application process based on the recognition result of the object recognition process from the feature extraction and object classification processing module 193. After performing the predetermined application process, the cloud sensor application 123 notifies the resource manager 121 of a resource release request.

[0135] In step S140, the resource manager 121 receives the resource release request from the cloud sensor application 123, stops the execution of each module of the ObjDetectionEAS 122, and releases the resources. As a result, the execution of the transport and decoding processing module 191, the candidate area determination processing module 192, and the feature extraction and object classification processing module 193 is stopped, and the resources are released.

[0136] In step S141, before the transport and decode processing module 191 stops its own execution, it notifies the image frame encoder module 152 to stop transferring the encoded image stream.

[0137] When the user device 111 does not have the DVS 31 installed, data processing for object recognition is performed as described above.

[0138] <5. Resource management by resource manager> <Resource configuration example> FIG. 14 is a diagram showing an example of the configuration of resources managed by the resource manager 121. As shown in FIG.

[0139] The resource manager 121 manages application resources so that the ObjDetectionEAS 122 operates in an edge environment (Edge Data Network). More specifically, the resource manager 121 manages the resources of the transport and decoding processing module 191, the candidate area determination processing module 192, and the feature extraction and object classification processing module 193.

[0140] The resources of the transport and decoding processing module 191 include the CPU time slots and memory necessary for executing the transport processing and the decoding processing. The resources of the candidate region determination processing module 192 include the CPU time slots and memory necessary for executing the candidate region determination processing. The resources of the feature extraction and object classification processing module 193 include the CPU time slots and memory necessary for executing the feature extraction processing and the object classification processing.

[0141] Also, the resource manager 121 manages the network / transport resources for transferring data between the ObjDetectionEAC133 and the ObjDetectionEAS122. Examples of such network / transport resources include a network based on a mobile communication network of a 5G line (hereinafter referred to as a 5G network), and a network based on an all-optical network (hereinafter referred to as an IOWN network) proposed by IOWN Global Forum, Inc.

[0142] <Management of Application Resources in the Absence of DVS> FIG. 15 is a diagram for explaining the life cycle management of the application resources of the ObjDetectionEAS122 when the DVS 131 is not implemented in the user device 111.

[0143] When the ObjDetectionEAC133 detects the generation of an I frame, in step S161, it transmits an I frame transfer timing notification to the transport and decoder resource management module 171 of the resource manager 121.

[0144] Upon receiving the I-frame transfer timing notification, transport and decoder resource management module 171 reserves the CPU time slots and memory required for transport processing, decoding processing, and candidate area determination processing, and starts each module in step S162. This starts transport processing module 191A, decoding processing module 191B, and candidate area determination processing module 192. In Figure 15, transport and decoding processing module 191 is shown divided into transport processing module 191A and decoding processing module 191B.

[0145] The candidate area determination processing module 192 executes the candidate area determination process using the baseband image data transferred and decoded from the ObjDetectionEAC 133. In the candidate area determination process, a candidate area may be determined using a single image frame, or may be determined using object transitions across multiple image frames. When multiple image frames are used, a delay occurs in the process because it is necessary to wait until multiple baseband image frames are generated.

[0146] In step S163, the candidate area determination processing module 192 notifies the feature extraction and classification processing resource management module 172 of candidate area determination information for the detected object.

[0147] Feature extraction and classification processing resource management module 172 receives object candidate area determination information from candidate area determination processing module 192. In step S164, feature extraction and classification processing resource management module 172 reserves CPU time slots and memory required to execute feature extraction processing and classification processing for each candidate area, and activates feature extraction and object classification processing modules 193 for the number of candidate areas.

[0148] When a resource release request is notified from the cloud sensor application 123, the resource manager 121 stops the execution of each module of the ObjDetectionEAS 122 and releases the secured CPU time slots and memory. Before stopping its own execution, the transport processing module 191A notifies the ObjDetectionEAC 133 to stop the transfer of the encoded image stream.

[0149] <Management of Application Resources in the Case of DVS> FIG. 16 is a diagram for explaining the life cycle management of the application resources of the ObjDetectionEAS 122 when the DVS 131 is implemented in the user device 111.

[0150] In step S181, the ObjDetectionEAC 133 detects a candidate region of a new object that has entered the imaging range and notifies the feature extraction and classification processing resource management module 172 of the resource manager 121 of the candidate region determination information of the object.

[0151] In step S182, based on the candidate region determination information of the object notified from the ObjDetectionEAC 133, the feature extraction and classification processing resource management module 172 secures the CPU time slots and memory necessary to execute the feature extraction and object classification processing module 193 for each candidate region, and starts the feature extraction and object classification processing module 193 by the number of candidate regions. After startup, each feature extraction and object classification processing module 193 enters a standby state for the baseband image data.

[0152] When the ObjDetectionEAC 133 detects the generation of an I-frame, in step S183, it transmits an I-frame transfer timing notification to the transport and decoder resource management module 171 of the resource manager 121.

[0153] Upon receiving the I-frame transfer timing notification, transport and decoder resource management module 171 reserves the CPU time slots and memory required for transport processing and decoding processing and starts each module in step S184. This starts transport processing module 191A and decoding processing module 191B. In Figure 16, transport and decoding processing module 191 is shown divided into transport processing module 191A and decoding processing module 191B.

[0154] The decoded baseband image data transferred from the ObjDetectionEAC 133 is supplied to each standby feature extraction and object classification processing module 193. Since the candidate regions have been identified, once one image frame is supplied, feature extraction processing and object classification processing can be performed immediately.

[0155] The execution of each module is stopped and resources are released in the same manner as in FIG. 15 when the DVS 131 is not present.

[0156] As described above, when DVS131 is implemented, it is possible to activate the feature extraction and object classification processing module 193 in advance according to the number of candidate regions and immediately execute the feature extraction processing and object classification processing, compared to when DVS131 is not implemented. This makes it possible to dynamically secure resources when needed and execute object recognition processing at high speed.

[0157] <Network / Transport Resource Management> <For 5G networks> Figure 17 shows an example of the network / transport configuration of a 5G network.

[0158] A 5G network consists of UE, AN (Access Network), and core network. The core network of the 5G system adopts a service-based architecture (3GPP TS.23.501 System architecture for the 5G System (5GS)). This service-based architecture defines NFs (Network Functions), which are the functions of the core network, and NFs are connected to each other via a unified interface called the service-based interface.

[0159] The UE 221 is a user device (mobile terminal). Under mobility management by the AMF 211 and session management by the SMF 212, the UE 221 connects to an external data network (such as an ISP or a corporate network connected via VPN) via packet communication (communication by transfer of packet data units (PDUs) such as IP or Ethernet) to receive services.

[0160] The AN 222 is a wired or wireless network between the UE 221 and the core network.

[0161] The AMF 211 performs mobility management, authentication, authorization, etc. of the UE 221. The AMF 211 also controls the SMF 212. The SMF 212 performs session management for the UE 221.

[0162] A UPF (User Plane Function) 223 transfers user data. A DN (Data Network) 224 is an external network where an application server and the like are located.

[0163] The logical connection relationship between ObjDetectionEAC133, which is an application on UE221, and ObjDetectionEAS122, which is an application on DV224, is called a PDU session 225. Resources required to form this PDU session 225, such as wireless and wired transfer paths for transferring packets and computational resources required for processing transfer protocols, correspond to network / transport resources managed by resource manager 121.

[0164] Fig. 18 is a diagram illustrating life cycle management of network / transport resources when the network / transport resources are a 5G network. Fig. 18 corresponds to life cycle management of application resources when there is no DVS 131 shown in Fig. 15.

[0165] When ObjDetectionEAC 133 detects the generation of an I frame, an I frame transfer timing notification is sent to transport and decoder resource management module 171 of resource manager 121 in step S161.

[0166] When the transport and decoder resource management module 171 receives an I frame transfer timing notification, the module is started by reserving a CPU time slot and memory required for transport processing and decoding processing in step S162, as described in Fig. 15. At the same time, in step S162, the transport and decoder resource management module 171 reserves resources for the PDU session 225 via the AN222 and UPF223 for the 5G system via the 5G system API, which are required to transfer the encoded image stream from ObjDetectionEAC133 to ObjDetectionEAS122.

[0167] The resource manager 121 releases the resources of the PDU session 225 at the timing when it stops the execution of each module of the ObjDetectionEAS 122 and releases the secured CPU time slots and memory.

[0168] <In the case of the IOWN network> FIG. 19 shows a configuration example of the network / transport of the IOWN network.

[0169] In the IOWN network, it is assumed that the transport stack configuration of the virtual path 231 formed between the ObjDetectionEAC 133 and the ObjDetectionEAS 122 is as shown in FIG. 19.

[0170] The transport stack is configured in the order of Fiber Layer (space division multiplexing (SDM) or mode division multiplexing (MDM) in one fiber), wavelength division multiplexing (WDM) of the WDM Layer, and time division multiplexing (TDM) of the TDM Layer from the bottom layer side, and on top of that, as the upper layer transport, it is composed of an IP packet layer or a non-IP layer.

[0171] Sessions realized on this stack are basically established in a connection-oriented manner. That is, when a connection is set up, a virtual path 231 is formed between the sending side and the receiving side by GMPLS (Generalized Multi-Protocol Label Switch) (network resources are secured). In the case of the transfer of the encoded image stream described above, ObjDetectionEAC 133 is the sending side and ObjDetectionEAS 122 is the receiving side, and the sending side secures the virtual path 231 that satisfies delivery requirements such as priority. To secure the virtual path 231, RSVP (Resource reSerVation Protocol)-TE (Traffic Engineering) extensions for GMPLS, etc., communicated on the control plane, are used. The resources for configuring this virtual path 231 correspond to the network / transport resources managed by the resource manager 121.

[0172] Fig. 20 is a diagram illustrating the life cycle management of network / transport resources when the network / transport resources are an IOWN network. Fig. 20 corresponds to the life cycle management of application resources when there is no DVS 131 shown in Fig. 15.

[0173] When ObjDetectionEAC 133 detects the generation of an I frame, an I frame transfer timing notification is sent to transport and decoder resource management module 171 of resource manager 121 in step S161.

[0174] When transport and decoder resource management module 171 receives an I frame transfer timing notification, the module is started up by reserving the CPU time slots and memory required for transport processing and decoding processing in step S162, as described in Fig. 15. At the same time, in step S162, transport and decoder resource management module 171 reserves, via the IOWN system API, resources for virtual path 231 via the AN equivalent and UPF equivalent, which are required for transferring the encoded image stream from ObjDetectionEAC 133 to ObjDetectionEAS 122 for the IOWN system.

[0175] The resource manager 121 stops the execution of each module of the ObjDetectionEAS 122 and releases the resources of the virtual path 231 at the same time as releasing the reserved CPU time slots and memory.

[0176] <6. Example of additional configuration of tracking processing module> Next, as another example of the configuration of the data processing system 100, a configuration in which an application that performs tracking processing of a recognized object is added after the processing of ObjDetectionEAS 122 will be described.

[0177] FIG. 21 is a block diagram showing an example of the configuration of a data processing system 100 to which an application that performs tracking processing has been added.

[0178] 21 shows only a part of the data processing system 100 necessary for explaining the application that performs the tracking process, and redundant explanations will be omitted as appropriate. Fig. 21 shows a configuration example corresponding to the case where the DVS 131 is implemented in the user device 111.

[0179] 21, an ObjTrackingEAS 251 is added as an application for performing tracking processing. A workflow manager 252 and a workflow description 253 are also provided.

[0180] The ObjTrackingEAS 251 is supplied with baseband image data and object recognition processing results from the ObjDetectionEAS 122. The object recognition processing of the ObjDetectionEAS 122 must be performed before the tracking processing performed by the ObjTrackingEAS 251.

[0181] The ObjTrackingEAS 251 has a tracking processing module 271 that tracks the trajectories of objects detected in the object recognition process. The tracking processing module 271 detects how each object recognized in the ObjDetectionEAS 122 moves within the entire image and how it will move thereafter, and notifies the cloud sensor application 123 of the trajectory tracking results. The ObjTrackingEAS 251 runs in either an Edge environment or on the cloud.

[0182] The workflow manager 252 runs in an edge environment or on the cloud and manages the resources of each processing module. The workflow manager 252 is the name of the resource manager 21 described above, but has been changed to conform to the MPEG-I-NBMP framework (ISO 23090-8:2018 Information technology - Coded representation of immersive media - Part 8: Network Based Media Processing).

[0183] The workflow manager 252 includes a tracking processing resource management module 281, a resource adjustment / optimization management module 282, and other processing resource management modules, including the transport and decoder resource management module 171 and the feature extraction and classification processing resource management module 172 shown in FIG.

[0184] The tracking processing resource management module 281 secures and executes resources for the tracking processing module 271. Furthermore, upon receiving a resource release request from the cloud sensor application 123, the tracking processing resource management module 281 stops the execution of the tracking processing module 271 and releases the resources.

[0185] The resource adjustment / optimization management module 282 refers to the workflow description (NBMP-WD) 253 in which the attributes of each application are described, and adjusts and optimizes the execution location of each application (EAS).

[0186] That is, the resource adjustment / optimization management module 282 analyzes the workflow description (NBMP-WD) 253 to understand the resource load situation in the edge environment and the cloud, and determines whether to migrate an application with relatively relaxed latency requirements to another edge environment or cloud, and if so, to which destination.

[0187] In an edge environment where ObjDetectionEAS122 is running, the load of applications running there may increase, causing application resources (including network / transport resources) such as CPU time slots and memory in the edge environment to become constrained, making it difficult to execute a new ObjTrackingEAS251 in that edge environment. Furthermore, the tracking process performed by ObjTrackingEAS251 may not have as strict a processing delay requirement as the object recognition process. Because the processing delay requirement for ObjTrackingEAS251 is less strict than that for ObjDetectionEAS122, if the application resources required to execute ObjTrackingEAS251 can be secured in a surrounding edge environment or on the cloud, the resource adjustment / optimization management module 282 adjusts resources so that ObjTrackingEAS251 is moved to another environment for execution. The workflow description (NBMP-WD) 253 describes conditions such as "if the processing delay requirements are relaxed compared to other applications, it may be executed in other edge environments or clouds" for a given application (processing module) as application attributes.

[0188] The data processing flow including resource adjustment / optimization of the ObjTrackingEAS 251 that performs tracking processing will be described with reference to FIG.

[0189] First, in step S201, the DVS data processing module 151 of the ObjDetectionEAC 133 analyzes event data supplied from the DVS 131 and detects an object within the imaging range. The ObjDetectionEAC 133 determines a candidate area for the object and notifies the feature extraction and classification processing resource management module 172 and the tracking processing resource management module 281 of the workflow manager 252 of the determined candidate area information for the object.

[0190] In step S202, the feature extraction and classification processing resource management module 172 receives the object candidate area determination information and, based on the candidate area determination information, secures resources for and executes the candidate area determination processing module 192 of the ObjDetectionEAS 122. Also, the tracking processing resource management module 281 receives the object candidate area determination information and, based on the candidate area determination information, secures resources for and executes the tracking processing module 271 of the ObjTrackingEAS 251.

[0191] In step S203, the image frame encoder module 152 of the ObjDetectionEAC 133 encodes the baseband image data supplied from the FBS 132. At this time, the image frame encoder module 152 determines whether to generate an I-frame using a scene change detection algorithm. If the image frame encoder module 152 determines to generate an I-frame, it sends an I-frame transfer timing notification to the transport and decoder resource management module 171 of the workflow manager 252 before encoding the I-frame.

[0192] In step S204, transport and decoder resource management module 171 receives the I frame transfer timing notification from image frame encoder module 152, and secures and executes resources for transport and decode processing module 191 of ObjDetectionEAS 122. The resources for transport and decode processing module 191 are secured before an I frame of the encoded image stream is transferred from image frame encoder module 152 in step S208, which will be described later.

[0193] In step S205, one or both of the transport and decoding processing module 191 of the ObjDetectionEAS 122 and the tracking processing module 271 of the ObjTrackingEAS 251 detects that resources cannot be secured and executed due to an increased resource load, and notifies the resource adjustment / optimization management module 282.

[0194] In step S206, the resource adjustment / optimization management module 282 of the workflow manager 252 analyzes the workflow description 253 and determines whether to move the ObjTrackingEAS 251, which has looser latency requirements, to an edge environment other than the edge environment where execution was previously attempted, or to the cloud. The workflow description 253 may be provided by the ObjDetectionEAC 133 or by an entity that manages workflow descriptions of the service provider.

[0195] When the resource adjustment / optimization management module 282 determines that the ObjTrackingEAS 251 should be moved, it determines another edge environment or cloud to which the ObjTrackingEAS 251 should be moved, and notifies the tracking processing resource management module 281 of a command to move the ObjTrackingEAS 251.

[0196] In step S207, the tracking processing resource management module 281 receives a move command for the ObjTrackingEAS 251 and the destination environment from the resource adjustment / optimization management module 282. Then, the tracking processing resource management module 281 secures (re-secures) resources for the tracking processing module 271 of the ObjTrackingEAS 251 and executes the resource in the specified destination environment. The ObjTrackingEAS 251 is executed in the re-secured environment.

[0197] In step S208, the image frame encoder module 152 of the ObjDetectionEAC 133 uplinks (transmits) an encoded image stream obtained by encoding the captured baseband image data to the transport and decode processing module 191 of the ObjDetectionEAS 122.

[0198] In step S209, the transport and decode processing module 191 of the ObjDetectionEAS 122 transports and decodes the encoded image stream. Baseband image data of the entire imaging range obtained by the decoding is supplied to the tracking processing module 271 of the ObjTrackingEAS 251.

[0199] In step S210, the feature extraction and object classification processing module 193 of the ObjDetectionEAS 122 executes feature extraction processing and object classification processing for each candidate area in parallel based on the baseband image data and the candidate area determination information of the object. The feature extraction and object classification processing module 193 notifies the tracking processing module 271 and the cloud sensor application 123 of the object classification processing result, i.e., the object recognition result.

[0200] The tracking processing module 271 executes tracking processing to track the trajectory of an object in an image, and notifies the cloud sensor application 123 of the tracking result of the trajectory.

[0201] As described above, the resource adjustment / optimization management module 282 can adjust and optimize resources by referring to the workflow description 253 and migrating applications with looser latency requirements to other edge environments or the cloud depending on the resource load situation.

[0202] FIG. 23 shows an example of the structure of the workflow description 253.

[0203] The workflow description 253 includes a General Descriptor, an Input Descriptor, an Output Descriptor, a Processing Descriptor, and a Requirements Descriptor. Of these, the Requirements Descriptor has an attribute "relativeProcessingDelayAcceptable." "relativeProcessingDelayAcceptable" can specify whether or not a processing delay is acceptable by using True or False.

[0204] <7. Summary> The data processing system 100 transfers image data acquired by a user device 111, which is a client-side device, to a network (cloud) 112, and executes object recognition processing on the network 112. The recognition results of the object recognition processing are sent to a cloud sensor application 123 and used for predetermined application processing.

[0205] Object recognition processing and other processes are highly demanding, and unnecessary processing must be reduced as much as possible. A system that transfers image data acquired by a user device 111 to a network 112, including an edge environment and the cloud, and constantly runs object recognition processing without knowing whether a new object has been added, wastes resources. Recognition applications with strict latency requirements are expected to increase in the future, and resource depletion and energy consumption may become major problems. Therefore, a method is needed that can dynamically secure resources as needed.

[0206] In the above-described data processing systems 1 and 100, resources required for object recognition processing are secured in an edge environment on the network when an I-frame is generated due to a scene change in a video. In other words, resources are secured when a new object is detected corresponding to a scene change, and object recognition processing begins. This allows resources to be dynamically secured sequentially as needed.

[0207] If the client device is equipped with a DVS (DVS31 or DVS131) ​​that outputs temporal brightness changes of optical signals as event data, it can detect new objects at a time granularity that cannot be detected with frame-based image data, so resources can be secured more quickly compared to when only a frame-based image sensor (FBS32 or FBS132) is used.In addition, since it can detect the number of new objects, resource load can be predicted, enabling more appropriate resource preparation.

[0208] <8.Other use case examples> This technology can be applied to processes other than the object recognition process in images described above. For example, this technology can be applied to medical and healthcare systems that use biological information measured in real time by body sensors.

[0209] A body sensor network is a type of sensor network made up of sensors placed on the surface of the body (wearable) and inside the body (implant).Recently, these sensors are connected to external monitoring / analysis applications via mobile networks or hospital networks (local or public 5G networks), and medical and healthcare systems based on real-time biometric information such as electrocardiograms, arterial oxygen saturation, and body temperature are rapidly becoming widespread.

[0210] For example, a medical application of DVS has been presented: the detection of acute or chronic diseases by measuring the flow rate and concentration of red blood cells in the microcirculation (the capillary network and its afferent and efferent arterioles and venules) using an event-based sensor. In the future, this technology could be applied to emergency treatment systems using image sensors such as frame-based sensors or event-based sensors to detect and recognize not only the flow of red blood cells within blood vessels but also foreign bodies (objects with shapes or colors different from red blood cells), and if they are dangerous, to immediately take action (such as trapping or crushing). For example, red blood cells flowing through blood vessels of a certain diameter typically move within the vessel with a nearly uniform shape. Therefore, even if a new red blood cell enters the imaging range, it is not detected by the encoder's scene change detection system. Instead, the block matching used in motion prediction considers it to be the "movement" of an object, and no scene change occurs (it is processed as a P-frame or B-frame). On the other hand, if a foreign body such as a detached thrombus enters the imaging range, it is detected as a scene change by the encoder. In such cases, it is necessary to immediately detect and recognize that it is a "foreign body" different from red blood cells and trigger emergency treatment according to the nature of the foreign body. Because processes like detecting intravascular foreign bodies do not occur frequently, constantly performing foreign body recognition processing based on image sensor frame images would be a huge waste of resources. As biometric sensor networks become more widely deployed in the future along with the spread of local 5G networks within each hospital, constantly recognizing and processing the massive amounts of image frame data constantly collected from patients' body sensors and enabling emergency response would require excessive network and computing resources to be secured and operated, resulting in constant waste of enormous amounts of power and energy. By applying this technology, the necessary resources can be secured in a timely manner only when a foreign body is likely to be detected, and can be released immediately after processing is complete.

[0211] This technology will be of great help not only to medical sensor networks but also to solving the energy consumption problem in data centers and to efficiently utilize resources for sensor applications in the cloud, as massive sensor networks are deployed across various industrial fields.

[0212] <9. Computer configuration example> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes microcomputers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0213] FIG. 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0214] In the computer, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are interconnected by a bus 304.

[0215] An input / output interface 305 is further connected to the bus 304. To the input / output interface 305, an input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected.

[0216] The input unit 306 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 307 includes a display, speaker, output terminal, etc. The storage unit 308 includes a hard disk, RAM disk, non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0217] In the computer configured as above, the CPU 301 performs the above-described series of processes by, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program. The RAM 303 also stores data necessary for the CPU 301 to execute various processes as needed.

[0218] The program executed by the computer (CPU 301) can be provided by being recorded on a removable recording medium 311 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0219] In a computer, the program can be installed in the storage unit 308 via the input / output interface 305 by inserting the removable recording medium 311 into the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be installed in the ROM 302 or the storage unit 308 in advance.

[0220] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0221] In addition, in this specification, the steps described in the flowcharts may be performed chronologically in the order described, but they do not necessarily have to be processed chronologically, and may be performed in parallel or at any necessary timing, such as when a call is made.

[0222] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0223] For example, in the above-described embodiment, an application that is configured with a plurality of modules may be configured with a single module, or may be further divided into a number of modules.

[0224] Each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0225] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0226] The present technology can have the following configurations. (1) a management module that secures resources on the network required for object recognition processing of image data transferred from the image sensor in accordance with the timing of I frame generation by the image sensor; A data processing device comprising: (2) The management module reserves the resources at the timing of generating an I-frame due to a scene change detected by an encoder that encodes the image data. The data processing device according to (1) above. (3) The management module executes an object recognition processing application using the reserved resources. The data processing device according to (1) or (2). (4) the object recognition process includes a candidate area determination process, The management module allocates the resources corresponding to the number of candidate areas detected in the candidate area determination process. The data processing device according to (3) above. (5) an event sensor is mounted on the device having the image sensor and outputs a temporal change in luminance of the optical signal as event data; The management module secures the resource upon detecting a new object corresponding to the I-frame generation based on the event data. The data processing device according to any one of (1) to (4). (6) The management module reserves the resources at a timing when the candidate area determination information of the new object is received from the event sensor. The data processing device according to (5) above. (7) the new object candidate region determination information includes the number of the new object candidate regions; The management module reserves the resources corresponding to the number of candidate regions for the new object. The data processing device according to (6) above. (8) The management module further reserves resources on the network necessary for a tracking process for tracking an object detected in the object recognition process. The data processing device according to any one of (1) to (7). (9) The management module further adjusts the execution location of the application according to the resource load status and processing delay requirements. The data processing device according to any one of (1) to (8). (10) The management module releases the secured resource based on the resource release request. The data processing device according to any one of (1) to (9). (11) a data processing device, At the timing of I-frame generation by the image sensor, resources necessary for object recognition processing of image data transferred from the image sensor are secured on the network. Data processing methods. (12) a client device that transfers image data generated by the image sensor to a network; a management module that secures resources necessary for object recognition processing of the image data on the network in accordance with the timing of I frame generation by the image sensor; A data processing system comprising: (13) The client device also includes an event sensor that outputs a temporal change in luminance of the optical signal as event data; The management module reserves the resource when a new object corresponding to the I-frame generation is detected based on the event data. The data processing system according to (12) above. [Explanation of symbols]

[0227] 1: Data processing system, 11: Client device, 12: Network, 21: Resource manager, 22: Object recognition application, 23: Cloud sensor application, 32: Frame-based sensor, 100: Data processing system, 111: User device, 112: Network, 121: Resource manager, 123: Cloud sensor application, 151: DVS data processing module, 152: Image frame encoder module, 171: Decoder resource management module, 172: Classification processing resource management module, 191: Decoding processing module module, 191A: transport processing module, 191B: decoding processing module, 192: candidate area determination processing module, 193: object classification processing module, 225: PDU session, 231: virtual path, 252: workflow manager, 253: workflow description, 271: tracking processing module, 281: tracking processing resource management module, 282: resource adjustment / optimization management module, 301: CPU, 302: ROM, 303: RAM, 306: input unit, 307: output unit, 308: memory unit, 309: communication unit, 310: drive

Claims

1. A data processing device connected via a network to a client device having at least an image sensor, a management module that, in response to a notification from the client device regarding the generation of an I frame by the image sensor, secures resources necessary for object recognition processing of image data of a moving image generated by the image sensor and transferred from the client device in a predetermined device on the network; Equipped with The resources include at least the time slots and memory of the CPU of the predetermined device. Data processing device.

2. The management module secures the resource in response to the notification in response to the timing when an encoder that encodes the image data detects a scene change.

2. The data processing device according to claim 1.

3. The management module executes an object recognition processing application using the secured resources.

2. The data processing device according to claim 1.

4. the object recognition process includes a candidate area determination process, The management module allocates the resources corresponding to the number of candidate areas detected in the candidate area determination process.

4. The data processing device according to claim 3.

5. the client device having the image sensor is equipped with an event sensor that outputs a temporal change in luminance of an optical signal as event data; The management module secures the resources in response to the notification of the timing of detecting a new object corresponding to the I-frame generation based on the event data.

2. The data processing device according to claim 1.

6. The management module reserves the resource at a timing when the candidate area determination information of the new object is received from the event sensor as the notification.

6. A data processing device according to claim 5.

7. the new object candidate region determination information includes the number of the new object candidate regions; The management module reserves the resources corresponding to the number of candidate regions for the new object.

7. The data processing device according to claim 6.

8. The management module further reserves resources necessary for a tracking process for tracking an object detected in the object recognition process in a predetermined device on the network.

2. The data processing device according to claim 1.

9. The management module further adjusts the execution location of the object recognition processing application or other applications scheduled to be executed according to the load status of the resource and the processing delay requirements.

2. The data processing device according to claim 1.

10. The management module releases the secured resource based on the resource release request.

2. The data processing device according to claim 1.

11. A data processing device connected via a network to a client device having at least an image sensor, and in response to a notification from the client device regarding the generation of an I frame by the image sensor, reserving resources necessary for object recognition processing of image data of a moving image obtained by the image sensor and transferred from the client device in a predetermined device on the network; The resources include at least the time slots and memory of the CPU of the predetermined device. Data processing methods.

12. A client device having at least an image sensor and transferring image data of a moving image generated by the image sensor to a network; a data processing device including a management module that, in response to a notification from the client device regarding the generation of an I frame by the image sensor, secures resources required for object recognition processing of the image data in a predetermined device on the network; Equipped with The resources include at least the time slots and memory of the CPU of the predetermined device. Data processing system.

13. The client device also includes an event sensor that outputs a temporal change in luminance of the optical signal as event data; The management module secures the resources in response to the notification in accordance with the timing at which a new object corresponding to the I-frame generation is detected based on the event data.

13. The data processing system of claim 12.

Images