Relay device
The relay device addresses processing delays in the pre-judgment processing system by managing packet transmission and certainty determination, optimizing data flow and reducing storage server reliance, thereby enhancing processing speed and throughput.
Patent Information
- Application Number
- PCT/JP2023/018414
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-21
AI Technical Summary
The existing pre-judgment processing system experiences significant processing delays due to bottlenecks in storing and reading inference target data, which hampers the efficiency of the multilayer inference system.
A relay device connected to the pre-judgment processing server, equipped with a packet holding unit, certainty determination unit, and transmission control unit, manages packet transmission by holding inference target data and determining the certainty of inference results, thereby optimizing the flow of data to the subsequent inference processing system.
This configuration suppresses processing delays in the pre-judgment processing system by efficiently managing packet transmission and reducing the reliance on storage servers, enhancing the overall processing speed and throughput.
Smart Images

Figure JP2023018414_21112024_PF_FP_ABST
Abstract
Description
relay device
[0001] The present invention relates to a relay device such as a network switch.
[0002] Inference techniques are known that use AI (Artificial Intelligence) to infer the contents (e.g., objects appearing in an image) of inference target data (e.g., image data). Non-Patent Document 1 discloses a multi-layer inference system that includes an inference processing system that performs inference and a pre-processing system that constitutes a front stage of the inference processing system and performs pre-processing for the inference.
[0003] Takeharu Eda and five others, "Highly Efficient Event-Driven Inference Supporting AI Services in the IOWN Era," NTT Technical Journal, December 2020, pp. 16-22
[0004] A possible hardware configuration for realizing a pre-processing system is a combination of a network switch to which inference target data is input and multiple servers connected to the network switch. The multiple servers include a storage server that acquires and stores inference target data from a data supply source via the network switch, a pre-processing server that acquires the inference target data from the storage server, pre-processes the inference target data, and stores the pre-processed inference target data back in the storage server, and a server that acquires the pre-processed inference target data from the storage server and infers the content of the inference target data.
[0005] In the above configuration, when inference target data is supplied to a subsequent inference processing system, the inference target data must be read from the storage server. Therefore, storing the inference target data in the storage server and reading the inference target data from the storage server become bottlenecks, causing large processing delays in the pre-processing system.
[0006] An object of the present invention is to suppress processing delays in a pre-processing processing system.
[0007] In order to solve the above problems, the relay device of the present invention is a relay device that constitutes the front stage of an inference processing system that performs inference and is connected to a pre-processing processing server that performs pre-processing processing for the inference, and is equipped with a packet holding unit that holds one or more packets representing inference target data supplied from a data supply source, a certainty determination unit that acquires the certainty of the result of the inference of the content of the inference target data by the pre-processing processing server and determines whether the acquired certainty satisfies a specified condition, and a transmission control unit that performs processing to transmit the one or more packets held by the packet holding unit to the inference processing system when it is determined that the certainty satisfies the specified condition.
[0008] According to the present invention, processing delays in the pre-processing processing system are suppressed.
[0009] Fig. 1 is a block diagram showing the configuration of a system including a relay device according to an embodiment of the present invention, and Fig. 2 is a block diagram showing the configuration of the relay device of Fig. 1.
[0010] Hereinafter, an embodiment of the present invention and its modifications will be described with reference to the drawings.
[0011] As shown in FIG. 1 , the relay device 30 according to this embodiment is configured as part of a multi-layer inference system 10. The multi-layer inference system 10 infers the contents of image data as inference target data. The image data is supplied from a client computer (hereinafter simply referred to as a client) 91, which serves as a data source and is connected to the multi-layer inference system 10 via a communication network NW. The client 91 controls the operation of a surveillance camera 92, causes the surveillance camera 92 to take images, acquires image data obtained by the images, and transmits the acquired image data to the multi-layer inference system 10. The image data is sequentially transmitted to the multi-layer inference system 10 at predetermined intervals.
[0012] The multi-layer inference system 10 infers the content of each piece of image data. The content of the image data is the object depicted in the image represented by the image data. The multi-layer inference system 10 transmits the inference results, i.e., candidate names of the objects depicted in the image and their probabilities (e.g., "person: 90%, cat: 8%, dog: 2%), to the client 91. The client 91 sets the surveillance camera 92 to recording mode while receiving the inference result with the highest probability of a person. As a result, the surveillance camera 92 records only during the period when a person is captured on the surveillance camera 92.
[0013] The multi-layer inference system 10 comprises a pre-processing processing system 20 as the first layer and an inference processing system 60 as the subsequent second layer. The systems 20 and 60 are capable of communicating via a communication network NW. The pre-processing processing system 20 is located at a pre-processing execution center, and the inference processing system 60 is located at a data center. The pre-processing execution center may be a location where a client 91 and a surveillance camera 92 are located, in which case the pre-processing processing system 20 may be directly connected to the client 91 without going through the communication network NW.
[0014] The pre-processing system 20 infers the content of image data using a low-precision but lightweight (low-power consumption) inference model. This inference is hereinafter also referred to as pre-processing. When the certainty of the inference result of the pre-processing system 20 is low, the inference processing system 60 infers the content of the image data using a high-precision inference model. With this configuration, the inference processing system 60 operates in response to the occurrence of an inference result with low certainty as an event, so the multi-layer inference system 10 can also be said to be an event-driven system. The inference results are returned to the client computer 91. Each inference model is an AI model, and each inference is based on AI.
[0015] The pre-processing system 20 comprises a relay device 30 connected to a communication network NW, a plurality of pre- and post-processing servers 41 to 43 connected to the relay device 30, and a plurality of pre-processing servers 51 to 53 connected to the relay device 30. The pre- and post-processing servers 41 to 43 are also collectively referred to as pre- and post-processing servers 40. The pre-processing servers 51 to 53 are also collectively referred to as pre-processing servers 50.
[0016] The relay device 30 is made up of a network switch. Each of the servers 41 to 43 and 51 to 53 is made up of a server computer. At least two of the servers 41 to 43 and 51 to 53 may be made up of a single server computer. Also, at least one of the servers 41 to 43 and 51 to 53 may be made up of a combination of multiple server computers.
[0017] Image data from the client 91 is input to the pre- and post-processing server 40 via the relay device 30. The pre- and post-processing server 40 performs pre-processing on the image data. The pre-processing includes at least one image processing such as noise removal, binarization, edge enhancement, or quantization, and also includes feature point extraction. The pre-processed image data (feature point data in this case) is input to the pre-processing server 50 via the relay device 30. The pre-processing server 50 performs pre-processing, inference of the content of the image data from the client 91 using the above-mentioned lightweight inference model based on the pre-processed image data. The inference model is assumed to be prepared in advance by machine learning (the same applies to high-precision inference models). The pre-processing result, i.e., the inference result, is input to the pre- and post-processing server 40 via the relay device 30. The pre- and post-processing server 40 performs post-processing on the inference result. The post-processing includes deriving the confidence level of the inference result obtained by the pre-processing. The inference result and the confidence level are input to the relay device 30. The relay device 30 holds image data before preprocessing, and if the confidence level is low, transmits the image data to the downstream inference processing system 60 via the communication network NW. If the confidence level is high, the inference result is returned to the client 91. This preprocessing reduces the operating rate of high-precision inference models, thereby reducing power consumption, etc.
[0018] The processing performed by servers 40 and 50 may be performed by a processor such as a CPU (Central Processing Unit) provided in the server executing a program, or by an accelerator such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit) provided in the server, or may be performed in a distributed manner by both the former and the latter.
[0019] The pre- and post-processing servers 41 to 43 perform the same pre-processing and post-processing. The pre-processing servers 51 to 53 each perform the same pre-processing. The relay device 30 also determines to which of the pre- and post-processing servers 41 to 43 the image data before pre-processing and the inference results should be sent, and to which of the pre-processing servers 51 to 53 the image data after pre-processing should be sent, in other words, performs load balancing.
[0020] The image data, inference results, and certainty factors relayed by the relay device 30 are transmitted and received in the form of packets, more specifically, in one packet or divided into multiple packets.
[0021] The relay device 30 has a plurality of ports, either physically or virtually, connected to the communication network NW and the servers 41 to 43 and 51 to 53, respectively, and the packets are transmitted and received via these ports.
[0022] The relay device 30 includes a header analysis unit 31 that analyzes packets input from each port, and a packet processing unit 32 that is located after the header analysis unit 31 and processes packets from the header analysis unit 31. The header analysis unit 31 and the packet processing unit 32 are circuits configured as ASICs or FPGAs, or a combination of these, that are included in the relay device 30.
[0023] The header analysis unit 31 analyzes the header of the packet and, depending on the analysis result, outputs the packet to the packet processing unit 32 or discards the packet. As an example, the determination of whether to output or discard may be made based on any one of the combination of source and destination addresses included in the header, the destination address alone, status information, or any combination thereof. The status information is information indicating the state of the packet. Examples of status information include an initial state indicating that the packet is an image data packet from the client 91, a pre-preprocessing state indicating that the packet is subject to preprocessing, a pre-inference processing state indicating that the packet is subject to inference processing, and a pre-postprocessing state indicating that the packet is subject to postprocessing.
[0024] The packet processing unit 32 includes a mirroring unit 33, a packet holding unit 34, a status information rewriting unit 35, an alive monitoring unit 36, a forwarding unit 37, a confidence level determining unit 38, and a transmission control unit 39. The packet from the header analysis unit 31 is input to either the mirroring unit 33, the status information rewriting unit 35, or the confidence level determining unit 38, depending on the status information included in the header of the packet.
[0025] The mirroring unit 33 mirrors the packets from the header analysis unit 31 , and outputs one of the packets to the packet holding unit 34 and the other to the state information rewriting unit 35 .
[0026] The packet holding unit 34 is made up of a queue, which is a FIFO (First-In First-Out) type memory, and sequentially holds packets from the mirroring unit 33. The packet holding unit 34 holds the packet until a series of processes related to the packet, including pre-processing, inference processing, and post-processing, is completed. The packet holding unit 34 may be provided on a semiconductor memory external to the ASIC or FPGA.
[0027] The state information rewriting unit 35 rewrites the state information of the input packet to the next state and outputs it to the forwarding unit 37 .
[0028] The alive / dead monitoring unit 36 performs alive / dead monitoring for each of the servers 41 to 43 and 51 to 53. The alive / dead monitoring unit 36 transmits a signal or packet to each of the servers 41 to 43 and 51 to 53, and measures the period from the timing of transmission to the timing of reception of a response to the signal or packet. The alive / dead monitoring unit 36 determines the busyness level of each server based on this measured time, and outputs each level to the forwarding unit 37 as an alive / dead monitoring result. The longer the measured time, the higher the busyness level. The alive / dead monitoring unit 36 may also perform alive / dead monitoring using other methods.
[0029] The forwarding unit 37 forwards the input packet to one of the servers 41 to 43 and 51 to 53 connected to the relay device 30. The forwarding unit 37 determines the destination of the packet based on the status information in the packet header and the health check result from the health check unit 36, and transmits, or forwards, the packet to the determined destination server.
[0030] The certainty determination unit 38 receives as input a packet after post-processing performed by the pre- and post-processing server on the inference result by the pre-processing server 50. The packet stores the certainty derived in the post-processing. Depending on the certainty of the input packet, the certainty determination unit 38 instructs the transmission control unit 39 to transmit the image data packet held in the packet holding unit 34 to the inference processing system 60 or to return the inference result to the client 91.
[0031] The transmission control unit 39 outputs packets of image data or packets of inference results to the outside in accordance with the transmission instruction.
[0032] Here, the operation of the relay device 30 when image data is divided into a plurality of packets and transmitted from the client 91 to the relay device 30 will be described.
[0033] As a premise, the client 91 includes the IP address of the client 91 as the source address and the IP address of any one of the pre- and post-processing servers 41 to 43 as the destination address in the header of each packet (header with a data length defined by a predetermined communication protocol) into which the image data is divided. The client 91 also includes data indicating the initial state as the above-mentioned state information at the beginning of the payload of each packet. After the beginning of each payload, data for each portion of the divided image data is stored.
[0034] The relay device 30 has a function that allows the user to set how much of the input packet from the beginning to the end should be treated as a header, and the relay device 30 uses this function to treat the beginning of the packet as part of the packet header. This makes it possible to operate using state information in conventional communication protocols.
[0035] Each packet of image data received by the relay device 30 is sequentially input to the header analysis unit 31. The header analysis unit 31 sequentially analyzes each input packet. In this case, the source address and destination address included in the packet header are the IP address of the client 91 and the IP address of one of the pre- and post-processing servers 41 to 43, respectively. Therefore, the header analysis unit 31 determines that the header contains specific information (i.e., a combination of the two addresses) that specifies that the packet should be output to the subsequent packet processing unit 32, and sequentially outputs the packets to the packet processing unit 32. When the state information included in the header is the initial state, the header analysis unit 31 may determine that the header contains the initial state as the specific information, and transmit the packets sequentially to the packet processing unit 32.
[0036] As described above, the header analysis unit 31 changes the output destination of the packet depending on the state information included in the header. Here, since the state information is in the initial state, the header analysis unit 31 sequentially outputs the packets to the mirroring unit 33 of the packet processing unit 32.
[0037] The mirroring unit 33 sequentially mirrors each packet from the header analysis unit 31 and sequentially sends one of the packets to the packet holding unit 34. The packet holding unit 34 holds each packet in a FIFO. The packets are held until a later operation of the transmission control unit 39. The mirroring unit 33 sequentially sends the other of the mirrored packets to the state information rewriting unit 35.
[0038] The state information rewriting unit 35 rewrites the state information contained in the header of the packet sent from the mirroring unit 33 from the initial state to the next pre-processing state, and sends the rewritten packet to the forwarding unit 37 in sequence.
[0039] Since the status information in the header of each packet sent from the status information rewriting unit 35 is the pre-pre-processing state, the forwarding unit 37 determines the pre- and post-processing servers 41 to 43 as candidate destinations for the packets. Thereafter, the forwarding unit 37 determines the pre- and post-processing server that is least busy as determined by alive / dead monitoring from the alive / dead monitoring unit 36 as the destination of each packet. Here, it is assumed that the pre- and post-processing server 41 has been determined as the destination. The forwarding unit 37 rewrites the destination address in the header of each packet to the IP address of the pre- and post-processing server 41, and sequentially forwards each packet to the port to which the pre- and post-processing server 41 is connected. As a result, the packets are sent to the pre- and post-processing server 41. Determining the destination based on alive / dead monitoring achieves load balancing.
[0040] The pre- and post-processing server 41 at the destination of each packet restores the image data divided into each packet from the payload data of each packet from the forwarding unit 37, and performs pre-processing on the image data. The pre- and post-processing server 41 divides the pre-processed image data into multiple packets and returns them to the relay device 30. The packet header stores the IP address of the pre- and post-processing server 41 as the source address, the IP address of any server 50 among the pre-processing servers 51 to 53 as the destination address, and the pre-inference processing state inherited from the packet from the forwarding unit 37 as state information.
[0041] Each packet received by the relay device 30 from the pre- and post-processing server 40 is input sequentially to the header analysis unit 31. The header analysis unit 31 sequentially analyzes each input packet. Here, the source address and destination address included in the packet header are the IP address of one of the servers 41 to 43 and the IP address of one of the pre-processing processing servers 51 to 53, respectively, so the header analysis unit 31 determines that the header contains the specific information and outputs the packets sequentially to the packet processing unit 32. When the state information included in the header is a pre-preprocessing state, the header analysis unit 31 may determine that the header contains the pre-preprocessing state as the specific information and output the packets sequentially to the packet processing unit 32.
[0042] Since the state information in the header is in the pre-preprocessing state, the header analysis unit 31 outputs the packets in question to the state information rewriting unit 35 in sequence.
[0043] The state information rewriting unit 35 rewrites the state information included in the header of each packet from the header analysis unit 31 to the state before the next inference process, and sends the rewritten packets to the forwarding unit 37 in sequence.
[0044] Since the status information in the header of each packet sent from the status information rewriting unit 35 is the pre-inference processing state, the forwarding unit 37 determines the pre-processing processing servers 51 to 53 as candidate destinations for the packets. The forwarding unit 37 then determines, as the destination of each packet, the pre-processing processing server 50, among the pre-processing processing servers 51 to 53, that is least busy as determined by the alive / dead monitoring unit 36. Here, it is assumed that the pre-processing processing server 51 has been determined as the destination. The forwarding unit 37 rewrites the destination address in the header of each packet to the IP address of the pre-processing processing server 51, and sequentially forwards each packet to the port to which the pre-processing processing server 51 is connected. As a result, the packets are sent to the pre-processing processing server 51.
[0045] The pre-processing server 51, which is the destination of each packet, restores the pre-processed image data (feature data in this case) that was divided into each packet from the payload data of each packet from the forwarding unit 37, and performs inference (pre-processing) using the lightweight inference model based on the image data. The pre-processing server 51 divides the inference result data into multiple packets (a single packet may be sufficient depending on the amount of data; the same applies to other data and multiple packets) and returns them to the relay device 30. The packet header stores the IP address of the pre-processing server 51 as the source address, the IP address of any server 40 among the pre- and post-processing servers 41 to 43 as the destination address, and the pre-inference processing state inherited from the packet from the forwarding unit 37 as state information.
[0046] Each packet received by the relay device 30 from the pre- and post-processing server 41 is input sequentially to the header analysis unit 31. The header analysis unit 31 sequentially analyzes each input packet. Here, the source address and destination address included in the packet header are the IP address of one of the pre-processing servers 51 to 53 and the IP address of one of the servers 41 to 43, respectively, so the header analysis unit 31 determines that the header contains the specific information and outputs the packets sequentially to the packet processing unit 32. If the state information included in the header is a pre-inference processing state, the header analysis unit 31 may determine that the header contains the specific information and output the packets sequentially to the packet processing unit 32.
[0047] Since the state information in the header is in the pre-inference processing state, the header analysis unit 31 outputs the packets in question to the state information rewriting unit 35 in sequence.
[0048] The state information rewriting unit 35 rewrites the state information included in the header of each packet from the header analysis unit 31 to the state before the next post-processing, and sends the rewritten packets to the forwarding unit 37 in sequence.
[0049] Since the status information in the header of each packet sent from the status information rewriting unit 35 is the pre-post-processing state, the forwarding unit 37 determines the pre- and post-processing servers 41 to 43 as candidate destinations for the packets. Thereafter, the forwarding unit 37 determines the pre- and post-processing server that is least busy as determined by the alive / dead monitoring unit 36, among the pre- and post-processing servers 41 to 43, as the destination of each packet. Here, it is assumed that the pre- and post-processing server 41 has been determined as the destination. The forwarding unit 37 rewrites the destination address in the header of each packet to the IP address of the pre- and post-processing server 41, and sequentially forwards each packet to the port to which the pre- and post-processing server 41 is connected. As a result, the packets are sent to the pre- and post-processing server 41.
[0050] The pre- and post-processing server 41 at the destination of each packet restores the inference results divided by each packet from the payload data of each packet from the forwarding unit 37 and performs post-processing on the inference results. The post-processing includes deriving a certainty factor. The inference result includes candidates for objects appearing in the image and the probability of the candidates being correct. For example, if the candidates are a person, a cat, and a dog, the inference result would be "person: 90%, cat: 8%, dog: 2%." The certainty factor indicates the degree of variability in the probability. The higher the variability, the lower the certainty factor. The pre- and post-processing server 41 derives the certainty factor from the inference result using any method. The pre- and post-processing server 41 divides the inference results and their certainty factors into multiple packets and returns them to the relay device 30. The packet header stores the IP address of the pre- and post-processing server 41 as the source address, the IP address of the relay device 30 as the destination address, and the pre-post-processing state inherited from the packet from the forwarding unit 37 as state information. The confidence factor may be stored in the packet header.
[0051] Each packet received by the relay device 30 from the pre- and post-processing server 41 is input sequentially to the header analysis unit 31. The header analysis unit 31 sequentially analyzes each input packet. In this case, the source address and destination address included in the packet header are the IP address of one of the pre- and post-processing servers 41 to 43 and the IP address of the relay device 30, respectively, so the header analysis unit 31 determines that the header contains the specific information and outputs the packets sequentially to the packet processing unit 32. When the status information included in the header is the pre-post-processing status, the header analysis unit 31 may determine that the header contains the pre-post-processing status as the specific information and output the packets sequentially to the packet processing unit 32.
[0052] Since the header state information is in the pre-post-processing state, the header analysis unit 31 sequentially outputs the packets to the certainty determination unit 38. The certainty determination unit 38 restores the inference result and certainty from the payload of each input packet. The certainty determination unit 38 determines whether the restored certainty is equal to or greater than a predetermined threshold. If the certainty is included in the packet header, the certainty determination unit 38 extracts the certainty from the header.
[0053] If the certainty is equal to or greater than the threshold, the certainty determination unit 38 outputs the inference result to the transmission control unit 39 and instructs the transmission control unit 39 to transmit the inference result. Upon receiving this instruction, the transmission control unit 39 reads from the packet holding unit 34 multiple packets for one image data item held in the packet holding unit 34. At this time, each read packet is deleted from the packet holding unit 34. The transmission control unit 39 extracts the IP address of the client 91 as the source address included in the header of this packet, generates one or more packets of the inference result including the extracted IP address as the destination address, and outputs these to a port connected to the communication network NW. This allows an inference result with a high certainty to be returned to the client 91. If the certainty is included in the packet header, the certainty determination unit 38 outputs the payload including the inference result as is to the transmission control unit 39, and the transmission control unit 39 may generate a new packet by adding a header including the destination address and the like to the payload. This eliminates the need to analyze the payload.
[0054] If the confidence level is less than the threshold, the confidence level determination unit 38 determines that the reliability of the inference result is low, and therefore it is better for the inference processing system to perform a high-precision inference. In this case, it instructs the transmission control unit 39 to transmit the image data held by the packet holding unit 34. Upon receiving this instruction, the transmission control unit 39 reads from the packet holding unit 34 multiple packets for one image data item held by the packet holding unit 34. At this time, each read packet is deleted from the packet holding unit 34. The transmission control unit 39 rewrites the destination address included in the header of this packet to the IP address of any one of the devices 61 to 65 of the inference processing system 60 and outputs the packet. Note that the source address is used as is. As a result, the reply destination of the inference result is clearly indicated in the header.
[0055] The inference processing system 60 includes a relay device 61 , a front-end server 62 , a storage server 63 , an inference control server 64 , and an inference processing server 65 .
[0056] The relay device 61 is made up of a network switch, and the servers 62 to 65 are made up of one or more server computers.
[0057] Image data from the pre-processing system 20 is supplied to the front-end server 62 via a relay device 61. The front-end server 62 stores the image data in a storage server 63 and issues an inference request to an inference control server 64. The storage server 63 stores data using a non-volatile storage disk device such as a hard disk, an SSD (Solid State Drive), or a PCIe card. The inference control server 64 queues the inference request and causes the inference processing server 65 to perform inference (including pre-processing, etc.) based on the inference request. The inference here uses an inference model with higher accuracy than that used in the pre-processing system 20. The inference result is returned from the inference processing server 65 to the client 91 via the front-end server 62 and the relay device 61.
[0058] According to this embodiment, the relay device 30 connected to the pre-processing server 50 includes a packet holding unit 34 that holds multiple packets representing one image data, more specifically, multiple packets each representing one image data by its payload data. When the certainty of the inference result is low, the multiple packets held in the packet holding unit 34 are sent to the inference processing system 60 by changing the destination of their headers. This configuration allows the image data or packets to be held in the relay device 30, which is a network switch. This speeds up the reading of the image data or packets, and suppresses (i.e., eliminates or reduces) processing delays in the pre-processing system 20. Furthermore, since the image data can be transferred to the inference processing system 60 simply by changing the destination of the header, the transfer of image data to the inference processing system 60 can be minimized. Furthermore, since a storage server is not used to hold the image data, the use of a storage server does not worsen the worst-case processing delay or adversely affect throughput.
[0059] In particular, since the relay device 30 is a network switch, image data is stored in the part corresponding to the entrance / exit of the pre-processing processing system 20, so that the image data can be transmitted to the inference processing system 60 at a higher speed than when the image data is stored in a downstream server or the like, thereby suppressing processing delays in the pre-processing processing system 20.
[0060] Furthermore, by configuring the packet holding unit 34 as an ASIC, FPGA, or semiconductor memory of a network switch, data can be read at high speed, and processing delays in the pre-processing processing system 20 can be suppressed.
[0061] Furthermore, since the confidence determination unit 38 and the transmission control unit 39 are configured using one or more ASICs and / or one or more FPGAs, these processes are performed faster than software processing, etc., and processing delays in the pre-processing processing system 20 are suppressed.
[0062] Furthermore, by mirroring, the image data supplied to the inference processing system 60 is also held on the relay device 30 side, separate from the image data used by the servers 40 and 50. Therefore, the image data is quickly supplied to the inference processing system 60, and processing delays in the pre-processing processing system 20 are suppressed.
[0063] Furthermore, the forwarding unit 37 determines the packet destination, i.e., the forwarding destination, based on state information included in the packet header, which is different before and after preprocessing and before and after inference, so one type of packet is used for both image data and data for determining the forwarding destination. This eliminates the need for the server to issue an inference request or the like separately from the image data, thereby reducing processing delays that may occur due to the issuance of an inference request or the like.
[0064] Furthermore, since storage servers and the like are no longer necessary, the number of servers in the pre-processing processing system 20 can be reduced, reducing operational costs, introduction costs, and the like.
[0065] As a modification, the forwarding unit 37 may monitor errors such as packet loss during packet forwarding. This error monitoring is also performed by the forwarding unit 37 on an ASIC or FPGA, thereby speeding up the process and reducing processing delays.
[0066] Other states such as pre-processed state, inferred state, and post-processed state may be added to the state information. Also, an error code used for alive monitoring or the like may be stored in the packet header, and the alive monitoring result such as the degree of busyness may be stored. Information on the usage status of the destination server may also be stored.
[0067] The packet header may be a TCP header, and six bits of the reserved area may store the packet status, error code, and destination alive monitoring results. By using the reserved area of the TCP header, there is no risk of malfunction even if the packet is input into the same processing system as the conventional TCP protocol. Furthermore, by expressing information in bits, the ASIC or FPGA of the relay device 30 can process information using only bit operations, so hardware resources can be kept small. Power consumption is also reduced.
[0068] The subject of inference may be something other than image data. For example, it may be audio data. The content of the audio data to be inferred may be the content of a conversation.
[0069] When the certainty of the inference result exceeds a threshold in the pre-processing processing system 20, an event is detected, and the inference target data used to infer that inference result can be sent to the inference processing system 60 and used as the subject of inference using a high-precision inference model.
[0070] The packet holding unit 34 may, for example, hold one or more packets representing inference target data supplied from a data source, although the specific form thereof is not limited to the above. The data source does not have to be a client computer and may be various sensors, etc. The certainty determination unit 38 may, for example, acquire the certainty of the result of inference of the content of the inference target data by the pre-processing processing server 50 and determine whether the acquired certainty satisfies a predetermined condition, although the specific form thereof is not limited to the above. The certainty satisfies the predetermined condition when the certainty is equal to or greater than a threshold value or when the certainty is equal to or less than a threshold value. This depends on the form of the pre-processing processing. The transmission control unit 39 may, for example, perform processing to transmit one or more packets held by the packet holding unit 34 to the inference processing system 60 when it is determined that the certainty satisfies the predetermined condition, although the specific form thereof is not limited to the above.
[0071] The relay device 30 may be, for example, a router other than a network switch. The packet holding unit 34 may be configured, for example, as an ASIC, an FPGA, or a semiconductor memory. The confidence determining unit 38 and the transmission control unit 39 may also be configured as one or more ASICs and / or one or more FPGAs.
[0072] For example, the transmission control unit 39 may perform the above-mentioned process by rewriting the destination address included in the header of the one or more packets to the address of a device that constitutes the inference processing system 60. The device may be any device.
[0073] The mirroring unit 33 may, for example, mirror one or more packets representing the inference target data, and the specific form thereof is not limited to the above-mentioned form. The forwarding unit 37 may, for example, forward the inference target data to a processing server that performs pre-processing of inference (pre- and post-processing server in the above example), and the specific form thereof is not limited to the above-mentioned form. Note that pre-processing may not be necessary, or may be included in the inference. In such a case, the forwarding unit 37 may forward one of the mirrored packets (packets whose state information is in the initial state) to the pre-processing processing server 50.
[0074] The forwarding unit 37 may, for example, forward one or more packets representing the inference target data before inference preprocessing to a processing server that performs preprocessing (pre- and post-processing server in the above example), and may forward one or more packets representing the inference target data after preprocessing to the pre-processing processing server 50; specific aspects are not limited to the above. For example, the forwarding unit 37 may determine the forwarding destination of one or more packets before and after preprocessing based on information that is different in content before and after preprocessing and that is included in the one or more packets. The information does not have to be status information.
[0075] The present invention is not limited to the above-described embodiments and modifications. For example, the present invention includes various modifications to the above-described embodiments and modifications that can be understood by a person skilled in the art within the scope of the technical concept of the present invention. The configurations listed in the above-described embodiments and modifications can be combined as appropriate within a range that does not cause contradictions. In addition, any of the above-described configurations can be deleted.
[0076] (Supplementary Notes) Configurations that are examples of the above-mentioned embodiments and modifications are described below. (Supplementary Note 1) A relay device that constitutes the front stage of an inference processing system that performs inference and is connected to a pre-processing processing server that performs pre-processing for the inference, comprising: a packet holding unit that holds one or more packets representing inference target data (which may be data after pre-processing or data before pre-processing) supplied from a data source, a certainty determination unit that acquires a certainty of the result of the inference of the content of the inference target data by the pre-processing processing server and determines whether the acquired certainty satisfies a predetermined condition (becomes equal to or greater than a threshold value or is less than a threshold value), and a transmission control unit that performs processing to transmit the one or more packets held by the packet holding unit to the inference processing system when it is determined that the certainty satisfies the predetermined condition (including processing the inference target data represented by the one or more packets and transmitting to the inference processing system one or more packets representing number theoretic object data after processing). (Supplementary Note 2) The relay device according to Supplementary Note 1, wherein the relay device is a network switch, and the packet holding unit is configured in an ASIC, an FPGA, or a semiconductor memory. (Supplementary Note 3) The relay device according to Supplementary Note 1 or 2, wherein the certainty factor determination unit and the transmission control unit are configured in one or more ASICs and / or one or more FPGAs. (Supplementary Note 4) The relay device according to any one of Supplements 1 to 3, wherein the transmission control unit performs, as the processing, a process of rewriting a destination address included in a header of the one or more packets to an address of a device that constitutes the inference processing system. (Supplementary Note 5) The relay device according to any one of Supplements 1 to 4, further comprising: a mirroring unit that mirrors the one or more packets; and a forwarding unit that forwards one of the mirrored packets to the pre-processing processing server or a processing server connected to the relay device that performs pre-processing of the inference on the inference target data, and the packet holding unit holds the other of the one or more mirrored packets.(Supplementary Note 6) A relay device as described in any of Supplementary Notes 1 to 5, further comprising a forwarding unit that forwards one or more packets representing the inference target data before the inference pre-processing to a processing server that performs the pre-processing, and forwards one or more packets representing the inference target data after the pre-processing to the pre-processing processing server, wherein the forwarding unit determines the forwarding destination of the one or more packets before and after the pre-processing based on information contained in the one or more packets that has different content before and after the pre-processing.
[0077] 10... multi-layer inference system, 20... processing system, 30... relay device, 31... header analysis unit, 32... packet processing unit, 33... mirroring unit, 34... packet holding unit, 35... status information rewriting unit, 36... alive monitoring unit, 37... forwarding unit, 38... certainty determination unit, 39... transmission control unit, 40-43... pre- and post-processing servers, 50-53... pre-processing processing server, 60... inference processing system, 91... client computer, 92... surveillance camera, NW... communication network.
Claims
1. A relay device that constitutes the front stage of an inference processing system that performs inference and is connected to a pre-processing processing server that performs pre-processing processing for the inference, comprising: a packet holding unit that holds one or more packets representing inference target data supplied from a data supply source; a certainty determination unit that acquires the certainty of the result of inference of the content of the inference target data by the pre-processing processing server and determines whether the acquired certainty satisfies a specified condition; and a transmission control unit that performs processing to transmit the one or more packets held by the packet holding unit to the inference processing system when it is determined that the certainty satisfies the specified condition.
2. The relay device according to claim 1, wherein the relay device is a network switch, and the packet holding unit is configured in an ASIC, an FPGA, or a semiconductor memory.
3. The relay device according to claim 2, wherein the confidence level determination unit and the transmission control unit are implemented in one or more ASICs and / or one or more FPGAs.
4. The relay device according to claim 1, wherein the transmission control unit performs, as the process, a process of rewriting a destination address included in a header of the one or more packets to an address of a device constituting the inference processing system.
5. A relay device as described in claim 1, further comprising: a mirroring unit that mirrors the one or more packets; and a forwarding unit that forwards one of the mirrored packets to the pre-processing server or to a processing server connected to the relay device that performs pre-processing of the inference on the inference target data, wherein the packet holding unit holds the other of the one or more mirrored packets.
6. The relay device of claim 1, further comprising a forwarding unit that forwards one or more packets representing the inference target data before the inference pre-processing to a processing server that performs the pre-processing, and forwards one or more packets representing the inference target data after the pre-processing to the pre-processing processing server, wherein the forwarding unit determines a forwarding destination of the one or more packets before and after the pre-processing based on information contained in the one or more packets and having different contents before and after the pre-processing.
Citation Information
Patent Citations
Floating population artificial intelligence early warning system based on cloud computing
CN113225539A
Method for interconnecting server and clients, and router
JP2003143212A
Traffic information estimation device, computer program for estimating traffic information, and traffic information estimation method
JP2011113547A
Slot type resolution process
US11308281B1
