Ai-powered terminal, ai system, method and program

Abstract

To flexibly adapt to various applications without affecting a process that requires availability.SOLUTION: An AI-powered terminal is configured to execute image recognition processing using artificial intelligence. The AI-powered terminal includes: system software for controlling the whole of the AI-powered terminal; first software and a first AI model related to a function that requires availability, out of functions of the AI-powered terminal; and second software and a second AI model which can be updated through external download. The second software and the second AI model are virtually separated from the system software, the first software and the first AI model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an AI-equipped terminal, an AI system, a method, and a program. [Background technology]

[0002] In recent years, image recognition technology using artificial intelligence (AI) has been put to practical use. Such image recognition technology is being used in a variety of applications, such as detecting people or objects, monitoring building facilities, and even in digital transformation (DX) services.

[0003] When deploying image recognition technology in DX services, flexible services that can accommodate individual needs, challenges, environments, etc. are required. Therefore, it is desirable to be able to appropriately update functions related to DX services depending on the challenges and applications. On the other hand, for applications such as monitoring building facilities, it is required to minimize downtime and be able to operate steadily. For this reason, it is difficult to balance applications for DX services with applications that require facility integration. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-125414 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide an AI-equipped terminal, AI system, method, and program that can flexibly respond to a variety of uses without affecting processing that requires availability. [Means for solving the problem]

[0006] The AI-equipped terminal of the embodiment is an AI-equipped terminal capable of performing image recognition processing using artificial intelligence, and includes system software that controls the entire AI-equipped terminal, first software and a first AI model related to functions of the AI-equipped terminal that require availability, and second software and a second AI model that can be updated by downloading from an external source, and the second software and the second AI model are virtually separated from the system software, the first software, and the first AI model. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an equipment system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of an AI-equipped terminal according to an embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of the software structure of the AI-equipped terminal according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing another example of the software structure of the AI-equipped terminal according to the embodiment. [Figure 5] FIG. 5 is a block diagram illustrating an example of a configuration of the cloud server according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a processing result of the AI processing executed by the AI-equipped terminal according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a processing result of AI processing updated by processing of the cloud server according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a processing procedure in the AI-equipped terminal according to the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of a processing procedure in the cloud server according to the embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of a procedure for updating software and an AI model of an AI-equipped terminal according to an embodiment. [Figure 11] FIG. 11 is a schematic diagram showing a procedure for generating an output image in an AI-equipped terminal according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Example of equipment system configuration) 1 is a diagram illustrating an example of the configuration of an equipment system 1 according to an embodiment. As shown in FIG. 1, the equipment system 1 includes an AI-equipped terminal 10, an imaging device 11, a gateway 20, a cloud server 30, a management terminal 40, and equipment 50.

[0009] Of these components, at least the AI-equipped terminal 10, cloud server 30, and management terminal 40 are configured as computers equipped with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. (not shown). The facility equipment 50 also has a control unit (not shown) that controls the facility equipment 50. The control unit of the facility equipment 50 is also configured as a computer equipped with a CPU, ROM, RAM, etc.

[0010] The imaging device 11 is connected to the AI-equipped terminal 10 via a wired or wireless connection, and the cloud server 30 is connected to the management terminal 40 via a wired or wireless connection. The AI-equipped terminal 10, the cloud server 30, and the facility equipment 50 are connected to each other via, for example, a gateway 20.

[0011] With the above-described configuration, the facility system 1 of the embodiment can control various facility devices 50 based on the processing results of the AI-equipped terminal 10, and can accumulate the processing results of the AI-equipped terminal 10 in the cloud server 30 to visualize various data or use it for future predictions.

[0012] The equipment system 1 of the embodiment in which artificial intelligence (AI) technology is used in the AI-equipped terminal 10 and the cloud server 30 as described below is an example of an AI system.

[0013] The imaging device 11 is a camera that uses a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or the like.

[0014] The facility system 1 includes one or more imaging devices 11, and each imaging device 11 is connected to a corresponding AI-equipped terminal 10. However, the imaging device 11 may be incorporated into the AI-equipped terminal 10, and the two may be configured as an integrated unit.

[0015] The AI-equipped terminal 10 uses AI to process images input from the imaging device 11. As an example, the AI performs image recognition and other processing to detect people, objects, etc. in the image. The AI-equipped terminal 10 outputs the images input from the imaging device 11 and the results of processing these images.

[0016] The gateway 20 is a relay device that enables communication between the AI-equipped terminal 10, the cloud server 30, and the facility equipment 50.

[0017] Specifically, the gateway 20 sends the processing results output from the AI-equipped terminal 10 to the corresponding facility device 50. The gateway 20 also collectively sends images and processing results output from one or more AI-equipped terminals 10 to the cloud server 30. At this time, the gateway 20 matches the data sent from the AI-equipped terminal 10 with the communication interface of the facility device 50 or cloud server 30 that is the destination. As the communication interface of the cloud server 30, for example, HTTPS (Hypertext Transfer Protocol Secure) can be used.

[0018] Furthermore, the gateway 20 sends out software, an AI model, various settings, etc. for the AI-equipped terminal 10 that are output from the cloud server 30 to the AI-equipped terminal 10. At this time, the gateway 20 matches the data sent from the cloud server 30 with the communication interface of the AI-equipped terminal 10.

[0019] In addition, if the AI-equipped terminal 10, the cloud server 30, and the equipment 50 have a common communication protocol, or if data can be sent and received directly between these devices, the equipment system 1 does not need to be equipped with a gateway 20.

[0020] The cloud server 30 stores images and processing results from the AI-equipped terminal 10 input via the gateway 20. The cloud server 30 also performs AI processing on the stored data and stores the processing results. As an example, the AI analyzes the stored data, processes it into statistical data, and visualizes the data. The data stored in the cloud server 30 and subjected to AI processing by the cloud server 30 is output to a management terminal 40 or the like and is widely used, for example, for digital transformation services.

[0021] Furthermore, the cloud server 30 appropriately provides software for performing cloud processing, software for the AI-equipped terminal 10, and AI models for the cloud server 30 and the AI-equipped terminal 10 to the cloud server 30 itself and the AI-equipped terminal 10. As a result, the software and AI models of the cloud server 30 and the AI-equipped terminal 10 are appropriately updated.

[0022] In the facility system 1 of the embodiment, instead of the cloud server 30, an on-premise server having the above-mentioned various functions of the cloud server 30 can also be used.

[0023] The management terminal 40 monitors the images and processing results from the AI-equipped terminal 10 that are stored in the cloud server 30. The management terminal 40 also configures, operates, and monitors the cloud server 30. Furthermore, the management terminal 40 manages and utilizes data by extracting and downloading necessary parts or necessary periods of the processing results from the cloud server 30, and displaying the downloaded data in graphs.

[0024] The facility equipment 50 is lighting equipment, air conditioning equipment, other infrastructure equipment installed in a building, line equipment installed in a factory, etc. In the facility equipment 50, the above-mentioned control unit (not shown) performs control in accordance with the processing results from the AI-equipped terminal 10 input via the gateway 20. As an example, when the AI-equipped terminal 10 detects that a person has entered an area where the facility equipment 50 is installed, it performs control such as turning on the lights in that area, operating the air conditioning, and adjusting the temperature and air volume according to the number of people in the area.

[0025] (Example of the structure of an AI-equipped device) Next, a detailed configuration example of the AI-equipped terminal 10 of the embodiment will be described with reference to FIGS. 2 and 3.

[0026] 2 is a block diagram showing an example of the configuration of the AI-equipped terminal 10 according to the embodiment. As shown in Fig. 2, the AI-equipped terminal 10 includes a control unit 110, an input unit 120, and an image processing unit 130 as functional units.

[0027] These functional units are realized by, for example, the above-mentioned CPU (not shown) provided in the AI-equipped terminal 10 expanding a control program stored in ROM into RAM and executing it. However, some or all of the above functional units may be configured with dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0028] The control unit 110 manages and controls the entire AI-equipped terminal 10.

[0029] The input unit 120 accepts inputs such as AI models, software, and various settings transmitted from the cloud server 30 via the gateway 20. The input unit 120 may also be capable of accepting inputs from a storage device such as a USB (Universal Serial Bus) memory connected to the main body of the AI-equipped terminal 10.

[0030] The AI model acquired by the input unit 120 from the cloud server 30 is set in the AI processing unit 134 (described later) of the image processing unit 130. The software acquired by the input unit 120 from the cloud server 30 is set in the function unit (described later) of the image processing unit 130. This updates the AI model used in the AI processing unit 134 and the functions of the image processing unit 130.

[0031] The image processing unit 130 is a functional unit that performs various processes on the image input from the imaging device 11, and includes an image input unit 131, an image adjustment unit 132, an image processing unit 133, an AI processing unit 134, a synchronization unit 135, an image compression unit 136, a processing result output unit 137, and an image output unit 138.

[0032] The image input unit 131 receives input of images captured by the imaging device 11 via a communication USB using Ethernet or the like, and baseband image input. Images from the imaging device 11 are input to the image input unit 131, including frames at, for example, 30 to 60 frames per second.

[0033] However, if the imaging device 11 is configured integrally with the AI-equipped terminal 10, the image may be acquired directly by the image processing unit 130 without going through the image input unit 131. In this case, the image processing unit 130 does not need to be equipped with the image input unit 131.

[0034] The image adjustment unit 132 performs digital processing such as lens distortion correction, exposure and gain adjustment, and noise removal on the image acquired from the imaging device 11 to adjust the image quality to be suitable for object detection, thereby facilitating image processing by the AI processing unit 134.

[0035] The image processing unit 133 performs operations such as enlarging or reducing the image that has been subjected to various adjustments by the image adjustment unit 132, or cutting out a predetermined area, and performs conversion according to the intended use of the image, such as image processing by the AI processing unit 134 or storage in the cloud server 30. The image for AI processing is sent to the AI processing unit 134, and the image for the cloud server 30 is temporarily buffered.

[0036] The AI processing unit 134 performs image recognition processing using AI to detect people or objects in an image, identify areas, etc. These AI processes employ machine learning using techniques such as DNN (Deep Neural Network).

[0037] The AI processing unit 134 does not perform such processing on all frames of the image acquired from the imaging device 11, but can perform it on some frames that have been thinned out at predetermined intervals.

[0038] The synchronization unit 135 notifies the image compression unit 136 of frame information that can identify a frame on which AI processing has been performed among multiple frames of an image acquired from the imaging device 11. The frame information that identifies a frame on which AI processing has been performed is, for example, a timestamp such as the imaging time of the frame, or a sequence number.

[0039] The image compression unit 136 appropriately compresses the temporarily buffered images in order to reduce the amount of data when transmitting the images to the cloud server 30 via the gateway 20. At this time, the image compression unit 136 adds frame information notified by the synchronization unit 135. The image compression unit 136 sends the compressed images to the image output unit 138.

[0040] The processing result output unit 137 outputs the processing result by the AI processing unit 134 to the cloud server 30 and the facility device 50 via the gateway 20. The processing result by the AI processing unit 134 output to the facility device 50 is referenced by the control unit of the facility device 50 and used for various controls of the facility device 50.

[0041] The image output unit 138 outputs the image compressed by the image compression unit 136 and to which the frame information has been added, to the cloud server 30 via the gateway 20.

[0042] Among the functions and uses of the AI-equipped terminal 10, the function of providing data used by the cloud server 30 requires flexible response to the diverse issues, needs, and environments of each service, in order to enable data utilization in, for example, DX services. Therefore, in order to update the AI model used in AI processing and the functions of the AI-equipped terminal 10, it is preferable that the operating environment allows frequent updates of the AI model and software.

[0043] Furthermore, among the functions and uses of the AI-equipped terminal 10, functions that require cooperation with the facility equipment 50, such as the function of generating and outputting processing results used for controlling the facility equipment 50, require availability that allows constant operation with as little downtime as possible. On the other hand, the cooperation function with the facility equipment 50 does not often require frequent updates of the AI model and software.

[0044] Therefore, the AI-equipped terminal 10 of the embodiment has a software structure separated by virtualization technology for each function and purpose of the AI-equipped terminal 10. That is, for example, a functional part that provides data utilized by the cloud server 30 and a functional part that requires cooperation with the facility equipment 50 are virtually separated, and it is possible to update the AI model and software for the functional part for the cloud server 30 without affecting the functional part that requires availability for the facility equipment 50.

[0045] Some examples of virtualization of the software structure in the AI-equipped terminal 10 of the embodiment are shown in FIGS. 3 and 4.

[0046] 3 is a block diagram showing an example of the software structure of the AI-equipped terminal 10 according to the embodiment. In the example shown in Fig. 3, virtualization software is used to virtualize the software structure in the AI-equipped terminal 10.

[0047] That is, the AI-equipped terminal 10a shown in Figure 3 is equipped with, for example, a hard disk HD, and virtualization software HV is executed on hardware having a host operating system (HOS), thereby setting and managing the execution environment of guest operating systems GOS (GOSx, GOSe), applications APP (APPx, APPe), and AI models MD (MDx, MDe) for each virtual machine VM (VMx, VMe) on the virtualization software HV.

[0048] Of these, the guest operating system GOSx, application APPx, and AI model MDx executed on the virtual machine VMx are related to the function of providing data utilized by, for example, the cloud server 30. Furthermore, the application APPx and the AI model MDx are configured to be updateable as needed by software and an AI model provided by the cloud server 30.

[0049] On the other hand, the guest operating system GOSe, application APPe, and AI model MDe executed on the virtual machine VMe are involved in functions that require availability, such as controlling facility devices 50 in cooperation with them. The application APPe and AI model MDe executed on the virtual machine VMe are configured to be able to continue processing without being affected even when the application APPx and AI model MDx of the virtual machine VMx are updated.

[0050] 4 is a block diagram showing another example of the software structure of the AI-equipped terminal 10 according to the embodiment. In the example shown in Fig. 4, container technology is used to virtualize the software structure in the AI-equipped terminal 10.

[0051] That is, the AI-equipped terminal 10b shown in Figure 4 is equipped with, for example, a hard disk HD, and by running the container management software MG on hardware having the host operating system HOS, the host operating system HOS is virtualized and the execution environment of the application APP (APPx, APPe) and the AI model MD (MDx, MDe) is set and managed for each container CNT (CNTx, CNTe) on the container management software MG.

[0052] Of these, the application APPx and AI model MDx executed in the container CNTx are related to the function of providing data to be utilized by the cloud server 30, for example, and are configured to be appropriately updated by the software and AI model provided by the cloud server 30.

[0053] On the other hand, the application APPe and AI model MDe executed in the container CNTe are related to functions that require availability, for example, in cooperation with equipment 50, and are configured to be able to continue processing without being affected even when the application APPx and AI model MDx of the container CNTx are updated.

[0054] 3 and 4 is an example of system software that controls the entire AI-equipped terminal 10. Furthermore, the application APPe related to the function requiring availability is an example of first software, and the application APPx related to image recognition processing in the cloud server 30 is an example of second software. Furthermore, the AI model MDe related to the function requiring availability is an example of the first AI model, and the AI model MDx related to image recognition processing in the cloud server 30 is an example of the second AI model.

[0055] (Cloud server configuration example) Fig. 5 is a block diagram showing an example of the configuration of the cloud server 30 according to the embodiment. As shown in Fig. 5, the cloud server 30 includes, as functional units, an output unit 310, an input unit 320, an AI processing unit 331, a correction / interpolation processing unit 332, a data processing unit 333, an image decompression unit 334, an image processing unit 335, an interface 340, an AI model library 351, a software library 352, a time-series data accumulation unit 353, and an image data accumulation unit 354.

[0056] These functional units are realized, for example, by the above-mentioned CPU (not shown) included in the cloud server 30 expanding a control program stored in ROM into RAM and executing it. However, some or all of the above-mentioned functional units may be configured with dedicated hardware such as ASIC, FPGA, etc.

[0057] The output unit 310 outputs the AI models and software described below stored in the AI model library 351 and the software library 352, respectively, as well as various settings, to the AI-equipped terminal 10 via the gateway 20.

[0058] The input unit 320 accepts input of processing results and images output from the AI-equipped terminal 10 via the gateway 20.

[0059] The AI processing unit 331 performs image recognition processing using AI to detect people or objects in images acquired from the AI-equipped terminal 10, identify areas, etc. These AI processes apply machine learning using techniques such as DNN.

[0060] Here, the cloud server 30 performs more complex and advanced AI processing compared to the AI processing performed by the AI-equipped terminal 10. As an example, the AI-equipped terminal 10 performs human detection in an image captured by the imaging device 11. In response to such processing results of the AI-equipped terminal 10, the cloud server 30 performs more advanced AI processing, such as estimating the posture of the person, estimating the facial expression of the person, or distinguishing a mobile robot from among the objects detected as people by the AI-equipped terminal 10.

[0061] During AI processing by the AI processing unit 331, the correction / interpolation processing unit 332 estimates and corrects errors in the processing results of the AI-equipped terminal 10, and updates the processing results by the AI-equipped terminal 10 stored in the time-series data accumulation unit 353.

[0062] The data processing unit 333 extracts necessary portions or periods of time-series data and image data in accordance with a request from the management terminal 40, and converts the data into a format suitable for statistical data display or graphing.

[0063] The image decompression unit 334 decompresses (decodes) the image data. The image data decompressed by the image decompression unit 334 is sent to the image processing unit 335 for processing by the AI processing unit 331, or sent to the data processing unit 333 for output to the management terminal 40.

[0064] The image processing unit 335 performs conversion suitable for image processing by the AI processing unit 331 by enlarging or reducing the image expanded by the image expansion unit 334 and cutting out a predetermined area.

[0065] The image expanded by the image expansion unit 334 and processed by the image processing unit 335 is used in various AI processes by the AI processing unit 331.

[0066] The interface 340 is an interface for communication with the management terminal 40 .

[0067] The AI model library 351 stores AI models to be executed by the AI-equipped terminal 10 and the cloud server 30. From these AI models, an appropriate AI model is selected depending on the intended use of the AI-equipped terminal 10 and the cloud server 30, and is set in the AI-equipped terminal 10 and the cloud server 30 as appropriate.

[0068] The software library 352 stores software to be executed on the AI-equipped terminal 10 and the cloud server 30. From this software, appropriate software is selected according to the intended use of the AI-equipped terminal 10 and the cloud server 30, and is set in the AI-equipped terminal 10 and the cloud server 30 as appropriate.

[0069] The time-series data accumulation unit 353 stores the processing results acquired from the AI-equipped terminal 10 as time-series data. Also, the time-series data accumulation unit 353 stores the processing results that the AI processing unit 331 performs further AI processing on based on the processing results acquired from the AI-equipped terminal 10, and that the correction / interpolation processing unit 332 interpolates and corrects as necessary, as time-series data.

[0070] The image data storage unit 354 stores compressed images acquired from the AI-equipped terminal 10 as image data. These image data are linked with frame information that can identify individual frames as embedded data within the image or as metadata.

[0071] (Example of equipment system operation) Next, an operation example of the facility system 1 according to the embodiment will be described with reference to Fig. 2 and Fig. 5. In the operation example described below, the AI-equipped terminal 10 performs person detection, and the cloud server 30 performs person posture estimation.

[0072] As shown in Figure 2, the AI-equipped terminal 10 acquires an image using an imaging device 11 or a built-in image sensor, etc. The image adjustment unit 132 of the AI-equipped terminal 10 performs exposure, gain adjustment, noise removal, etc. on the acquired image to adjust the image quality to be suitable for processing by the AI processing unit 134.

[0073] The image processing unit 133 processes images for processing by the AI processing unit 134 or for output to the cloud server 30.

[0074] For example, for the AI processing unit 134, the image processing unit 133 can reduce the size of the image in accordance with the processing performance of the AI processing unit 134. In this case, to suppress image degradation, the image processing unit 133 may perform interpolation processing using a method such as bilinear or bicubic. Furthermore, for the cloud server 30, the image processing unit 133 can maintain the image at high resolution with minimal degradation.

[0075] The AI processing unit 134 detects people in the images by performing image recognition processing or the like using machine learning, such as DNN, on images sampled at predetermined intervals from among the images processed by the image processing unit 133. When detecting people in the images, the AI processing unit 134 performs processing such as assigning a bounding box that surrounds the area in the image in which a person appears with the smallest rectangle.

[0076] An example of the processing result of such AI processing is shown in Figure 6. Figure 6 is a diagram showing an example of the processing result of AI processing executed by the AI-equipped terminal 10 according to the embodiment. The processing result shown in Figure 6 includes frame information of both a sequence number and a timestamp. However, in actual processing, it is sufficient to include at least one of the sequence number and the timestamp.

[0077] The processing result output unit 137 outputs the processing result by the AI processing unit 134 to the cloud server 30 and the facility device 50. At this time, the processing result output unit 137 can output the processing result via, for example, the gateway 20. This makes it possible to match the output format with the interfaces of the cloud server 30 and the facility device 50, and also makes it possible to bundle outputs from multiple AI-equipped terminals 10 and send them to the cloud server 30.

[0078] The synchronization unit 135 notifies the image compression unit 136 of frame information such as the timestamp or sequence number of the image processed by the AI processing unit 134. Here, if a failure occurs in the image processing by the AI processing unit 134, the synchronization unit 135 also performs a process of selecting some frames obtained by thinning out all frames of the image acquired from the imaging device 11 at predetermined intervals.

[0079] The image compression unit 136 selects an image to compress from all frames of images acquired from the imaging device 11 based on the frame information notified by the synchronization unit 135. If a failure occurs in the image processing by the AI processing unit 134, the image compression unit 136 will compress the frame selected by the synchronization unit 135.

[0080] Furthermore, the image compression unit 136 compresses the image in a moving image format or a still image format depending on the bandwidth of the communication path and the capacity of images stored in the cloud server 30. When compressing the image, the image compression unit 136 embeds the frame information notified by the synchronization unit 135 in the image or adds it to the image as metadata.

[0081] However, the frame information of each image may be added to these images in advance at the timing when the images are acquired from the imaging device 11.

[0082] The image output unit 138 outputs the image compressed by the image compression unit 136 and to which the frame information has been added to the cloud server 30. The compressed image is output in a file format or by streaming, for example.

[0083] At this time, the image output unit 138 can output the processing result via, for example, the gateway 20. This allows the output format to be matched with the interface of the cloud server 30.

[0084] 5, the input unit 320 of the cloud server 30 acquires the processing result and the compressed image from the AI-equipped terminal 10. As described above, the image is transmitted using, for example, a file format or a streaming method.

[0085] The input unit 320 stores the processing results from the AI-equipped terminal 10 as time-series data in the time-series data storage unit 353, and stores the compressed images from the AI-equipped terminal 10 as image data in the image data storage unit 354. The time-series data storage unit 353 stores the processing results from the AI-equipped terminal 10, and the image data storage unit 354 stores the compressed images from the AI-equipped terminal 10.

[0086] The image data stored in the image data storage unit 354 is read out as needed and used for AI processing by the AI processing unit 331, or is output to the management terminal 40.

[0087] For processing by the AI processing unit 331, the image decompression unit 334 decompresses the image data to be processed by the AI processing unit 331. The image processing unit 335 performs conversion suitable for image processing by the AI processing unit 331 by enlarging or reducing the image decompressed by the image decompression unit 334, cutting out a predetermined area, etc.

[0088] The AI processing unit 331 performs image recognition processing, etc. on the image processed by the image processing unit 335 by machine learning using, for example, DNN, etc., and estimates the posture of the person in the image.

[0089] More specifically, the AI processing unit 331 refers to frame information such as a timestamp or sequence number assigned to the image data, and reads out time-series data corresponding to the image data from the time-series data storage unit 353. The AI processing unit 331 also acquires, from the read-out time-series data, coordinates or the like corresponding to an area such as a bounding box assigned to the image by the AI-equipped terminal 10 when detecting a person, and estimates the person's posture, such as walking, standing, sitting, or falling, at the coordinate part where the person is detected.

[0090] In this way, the AI-equipped terminal 10 only needs to perform, for example, posture estimation processing on the area where a person has been detected and identified in advance, thereby reducing the processing load on the cloud server 30.

[0091] However, due to the inference processing performance of the cloud server 30 or a lack of image transmission bandwidth, the cloud server 30 may not be able to perform AI processing on all images processed by the AI-equipped terminal 10. In this case, the AI processing unit 331 may further sample from all images processed by the AI-equipped terminal 10 at predetermined intervals and perform AI processing on those images.

[0092] In this way, AI processing may be performed only on some sampled images in the cloud server 30. In addition, since the AI-equipped terminal 10 performs AI processing on some frames that have been thinned out at predetermined intervals, for example, there is a possibility that errors may be included in the processing results.

[0093] During AI processing by the AI processing unit 331, if the processing result of the AI-equipped terminal 10 contains an error, the correction / interpolation processing unit 332 performs error correction, for example, by interpolating data between frames to be processed by estimation.

[0094] Various processes are performed by the AI processing unit 331, and the new processing results that have been error-corrected by the correction / interpolation processing unit 332 are stored anew in the time-series data accumulation unit 353. As a result, the processing results by the AI-equipped terminal 10 are updated to the processing results by the AI processing unit 331 as appropriate.

[0095] FIG. 7 shows an example of an updated processing result by the AI-equipped terminal 10 in FIG. 6. FIG. 7 is a diagram showing an example of an updated processing result of AI processing by the cloud server 30 according to an embodiment. The processing result shown in FIG. 7 also includes frame information of both a sequence number and a timestamp. However, in actual processing, at least one of the sequence number and the timestamp assigned to the image by the AI-equipped terminal 10 will be included.

[0096] The time series data stored in the time series data storage unit 353 is read out in response to a request from the management terminal 40, and is output to the management terminal 40 after being processed as appropriate.

[0097] At this time, the management terminal 40 specifies the AI-equipped terminal 10 corresponding to the time-series data and image data to be read, the output items, the output period, etc., and makes an output request to the cloud server 30. In response to the output request from the management terminal 40, the data processing unit 333 reads out the time-series data corresponding to the specified data from the time-series data accumulation unit 353, and sorts and rearranges the read data.

[0098] The data processing unit 333 sends the organized and rearranged time series data to the interface 340 together with the image data read from the image data storage unit 354. The interface 340 converts the time series data and image data output by the data processing unit 333 into a communication interface that is compatible with the management terminal 40, and outputs the data to the management terminal 40.

[0099] The time-series data that has been subjected to AI processing by the AI processing unit 331 and error-corrected and updated by the correction / interpolation processing unit 332 may be fed back to the facility equipment 50. For example, when the AI-equipped terminal 10 detects a person but it is an error, the control unit of the facility equipment 50 may turn off a light that was once turned on for the erroneously detected person, based on feedback from the cloud server 30. Furthermore, when the number of people detected by the AI-equipped terminal 10 is incorrect, the control unit of the facility equipment 50 may reset the air conditioning temperature that was set according to the incorrect number of people to the correct number of people, based on feedback from the cloud server 30.

[0100] (Example of equipment system processing) Next, examples of various processes in the facility system 1 of the embodiment will be described with reference to FIGS.

[0101] FIG. 8 is a flowchart showing an example of a processing procedure in the AI-equipped terminal 10 according to the embodiment.

[0102] As shown in FIG. 8, when the image input unit 131 acquires an image (step S101), the image adjustment unit 132 adjusts the image quality by adjusting the exposure, gain, and noise removal of the image (step S102).

[0103] Furthermore, the image processing unit 133 adjusts the size of the image, cuts it out, etc. so that it is suitable for AI processing or image output (step S103). That is, the image processing unit 133 reduces the image and cuts out a predetermined area for AI processing, and maintains the image at high resolution for output.

[0104] The image processed for AI processing is sent to the AI processing unit 134, where various AI processes are performed (step S104). The processing results by the AI processing unit 134 are output to the facility device 50 by the processing result output unit 137 (step S110).

[0105] Meanwhile, the synchronization unit 135 acquires frame information such as a timestamp or sequence number of the image to be processed by the AI processing unit 134, and performs synchronization processing so that the processing result by the AI processing unit 134 can later be associated with the target image (step S105).

[0106] Meanwhile, images processed for output by the image processing unit 133 are temporarily buffered. From these images, the image compression unit 136 selects, as output images, images that have been subjected to synchronization processing by the synchronization unit 135 (step S106). Furthermore, the image compression unit 136 embeds frame information in the selected images or adds the information to the images as metadata (step S107), and then compresses these images (step S108).

[0107] Image output unit 138 outputs the images to which frame information has been added and compressed by image compression unit 136 to cloud server 30. Furthermore, the processing results of these images in the process of step S110 are output to cloud server 30 by processing result output unit 137 (step S109).

[0108] This completes the processing in the AI-equipped terminal 10 of the embodiment.

[0109] FIG. 9 is a flowchart illustrating an example of a processing procedure in the cloud server 30 according to the embodiment.

[0110] As shown in FIG. 9, when the input unit 320 acquires a compressed image and a processing result from the AI-equipped terminal 10 (step S201), it stores the compressed image in the image data storage unit 354 (step S202) and stores the processing result in the time-series data storage unit 353 (step S203).

[0111] The image data storage unit 354 stores compressed images from the AI-equipped terminal 10 as image data, and the time-series data storage unit 353 stores processing results from the AI-equipped terminal 10 as time-series data.

[0112] When performing AI processing on image data, the image decompression unit 334 reads and decompresses the image data to be processed from the image data storage unit 354 (step S204). Furthermore, the image processing unit 335 processes the read image so that it is suitable for AI processing (step S205).

[0113] The AI processing unit 331 performs various AI processes using a machine learning method such as DNN on the image data that has been read from the image data storage unit 354, expanded, and further processed (step S206). At this time, the AI processing unit 331 reads and refers to the processing result of the AI-equipped terminal 10 for the image to be processed, which was stored in the time-series data storage unit 353 in the processing of step S303.

[0114] The correction / interpolation processing unit 332 corrects errors in the processing by the AI-equipped terminal 10 in parallel with the AI processing by the AI processing unit 331, and stores and updates the processing results by the AI processing unit 331 in the time-series data accumulation unit 353 (step S207).

[0115] In response to a request from the management terminal 40, the data processing unit 333 reads and processes the necessary parts and necessary periods of the processing results by the AI processing unit 331 from the time-series data storage unit 353, reads the corresponding image data from the image data storage unit 354 (step S208), and outputs them to the management terminal 40 via the interface 340 (step S209).

[0116] This completes the processing in the cloud server 30 of the embodiment.

[0117] FIG. 10 is a flowchart showing an example of a procedure for updating the software and AI model of the AI-equipped terminal 10 according to the embodiment.

[0118] As shown in FIG. 10, the input unit 120 of the AI-equipped terminal 10 downloads software or AI models output from the software library 352 or the AI model library 351 via the output unit 310 of the cloud server 30 as needed (step S301).

[0119] The control unit 110 updates the application APPx or AI model MDx of the image processing unit 130, which is separated from other functional units by virtualization and is related to the function of providing data to be utilized on the cloud server 30, with the downloaded software or AI model (step S302).

[0120] The control unit 110 also controls the entire AI-equipped terminal 10 to update the startup script according to the updated software or AI model (step S303), and then restarts virtualized elements such as virtual machines VMx or containers CNTx that are involved in the function of providing data utilized by the cloud server 30 (step S304). As a result, the startup script according to the updated software or AI model is executed (step S305).

[0121] This completes the update process in the AI-equipped terminal 10 of the embodiment.

[0122] (Overview) In recent years, technologies have been put into practical use to detect people or objects by equipping or connecting surveillance cameras with AI, or by outputting surveillance camera images to cloud servers capable of AI processing and performing image recognition using AI. Digital transformation services that utilize image recognition data, such as visualization and prediction of image recognition data, are also beginning to be developed. In this case, flexible service deployment becomes possible by using cloud servers with abundant hardware resources.

[0123] However, when AI-based image recognition processing is performed on a cloud server, etc., images must be transmitted from surveillance cameras, etc., making real-time processing difficult and creating the problem of securing bandwidth for the transmission path.

[0124] On the other hand, there is a trade-off between improving image recognition accuracy and performing sophisticated and complex processing on the one hand, and reducing the amount of calculations on the other. Therefore, for example, while an AI-equipped terminal can provide immediate response in controlling facility equipment, applying highly accurate algorithms to perform sophisticated and complex AI processing on the other hand increases the cost of the device, and also creates issues with installation space and heat dissipation, making it difficult to achieve sufficient calculation performance.

[0125] Furthermore, it is desirable to frequently update software and AI models in order to flexibly respond to digital transformation services, etc. However, if an AI-equipped device also has a function for linking with facility equipment such as surveillance cameras, frequent software and AI model updates are not desirable because they impair the availability of the AI-equipped device.

[0126] The AI-equipped terminal 10 of the embodiment comprises a host operating system HOS that controls the entire AI-equipped terminal 10, an application APPe and an AI model MDe related to functions that require availability among the functions of the AI-equipped terminal 10, and an application APPx and an AI model MDx that can be updated by downloading from an external source. The application APPx and the AI model MDx are virtually separated from the host operating system HOS, the application APPe, and the AI model MDe.

[0127] By providing the AI-equipped terminal 10 with such a software structure, it is possible to separate the execution environments for processes that require availability, such as linking with facility devices 50, from processes that require flexible services for utilizing data in DX services, etc. Therefore, it is possible to update the application APPx and the AI model MDx without affecting the processes that require availability, and it is possible to flexibly respond to DX services, etc. Therefore, it is possible to respond to a variety of uses while continuing processes that require availability.

[0128] According to the embodiment, the equipment system 1 includes an AI-equipped terminal 10 capable of executing image recognition processing using AI, and a cloud server 30 that verifies the processing results of the AI-equipped terminal 10 through the image recognition processing using AI and performs error correction.

[0129] This allows the AI-equipped terminal 10 to perform processes that require immediate response, such as control of the facility devices 50, while the cloud server 30 can perform processes that require advanced and complex processing but do not require immediate response, such as data analysis. Also, by having the cloud server 30 perform follow-up processing, the processing results of the AI-equipped terminal 10 can be corrected to correct data, thereby improving the reliability of the data.

[0130] According to the equipment system 1 of the embodiment, the cloud server 30 extracts a predetermined image from the images that have been subjected to image recognition processing by the AI-equipped terminal 10, performs verification processing, and if there is an error in the processing result of the AI-equipped terminal 10, corrects the error by interpolating data between the images that have been subjected to the verification processing. This makes it possible to infer the correct result from the processing result performed by sampling, and correct the processing result of the AI-equipped terminal 10.

[0131] According to the embodiment of the equipment system 1, the cloud server 30 includes a software library 352 storing software for the AI-equipped terminal 10 and the cloud server 30, and an AI model library 351 storing AI models for the AI-equipped terminal 10 and the cloud server 30, and is configured to be able to update the AI-equipped terminal 10 and the cloud server 30 with the above software or the above AI model.

[0132] As described above, the processing result by the AI-equipped terminal 10 is synchronized with the image to be processed, allowing seamless sharing of processing between the AI-equipped terminal 10 and the cloud server 30. Therefore, software and AI models can be executed seamlessly between the AI-equipped terminal 10 and the cloud server 30 without being aware of the execution platform. This makes it easy to optimize the entire system.

[0133] Furthermore, by applying virtualization technology to the AI-equipped terminal 10, the instances executed by the AI-equipped terminal 10 and the cloud server 30 can be shared.

[0134] According to the embodiment of the equipment system 1, the cloud server 30 acquires from the AI-equipped terminal 10, together with the processing result of the AI-equipped terminal 10, an image with frame information that can identify the image that was the subject of the image recognition processing by the AI-equipped terminal 10.

[0135] This allows the processing results of the AI-equipped terminal 10 to match the images used in that processing in the cloud server 30. Therefore, the processing results of the AI-equipped terminal 10 can be verified in the cloud server 30, and the AI-equipped terminal 10 and the cloud server 30 can work together to seamlessly share processing. Furthermore, even if there are limitations on the performance of the AI-equipped terminal 10, the system as a whole can execute complex, high-load processing.

[0136] In the above-described embodiment, the equipment system 1 performs AI processing on the images captured by the imaging device 11, but an AI system such as the equipment system 1 may also be capable of acquiring an audio signal from an audio sensor and performing voice recognition processing using AI processing, or may be capable of acquiring an amplitude signal due to vibration of equipment from a vibration sensor and performing fault diagnosis processing of the equipment using AI processing.

[0137] In the above-described embodiment, the processing result of the AI-equipped terminal 10 is output to the cloud server 30 separately from the compressed image to which frame information is added. However, the processing result of the AI-equipped terminal 10 may also be added to the output image to the cloud server 30. Such processing is shown in FIG. 11.

[0138] 11 is a schematic diagram showing the procedure for generating an output image in an AI-equipped terminal according to a modified example of the embodiment. As shown in FIG. 11, the image input unit 131 acquires an image including frames (n, n+1, n+2, n+3, n+4, n+5...) from the imaging device 11. The AI processing unit 134 performs AI processing such as image recognition on frames (n, n+2, n+4...) sampled at predetermined intervals from the acquired images, and generates a processing result.

[0139] Furthermore, frames (n, n+1, n+2, n+3, n+4, n+5...) acquired from the imaging device 11 are temporarily buffered, for example, in image buffer 130b in image processing unit 130. The image output unit 138 outputs frames (n, n+2, n+4...) to which synchronization processing has been performed by synchronization unit 135, compression has been performed by image compression unit 136, and the processing result of AI processing unit 134 has been added, to cloud server 30.

[0140] At this time, the synchronization unit 135 notifies the image compression unit 136 of the processing result of the AI processing unit 134 along with frame information regarding the image on which AI processing has been performed. The image compression unit 136 embeds the processing result of the AI processing unit 134 together with the frame information within the image, or adds it to the image as metadata, and then compresses the image. In addition, the processing result of the AI processing unit 134 may be added to the image using, for example, a VBI (Vertical Blanking Interval) superimposition technique that superimposes information on blank areas between frames, or may be superimposed on an ineffective area of the image.

[0141] As described above, by adding the processing result by the AI-equipped terminal 10 to the image that was the processing target, the processing result and the image can be synchronized and matched on the cloud server 30.

[0142] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0143] 1. Equipment system 10 AI-equipped devices 11 Imaging device 20 Gateway 30 Cloud Servers 40 Management terminal 50 Equipment 351 AI model library 352 Software Library APPe, APPx applications HOS Host Operating System MDe, MDx AI model

Claims

1. An AI-equipped terminal capable of performing image recognition processing using artificial intelligence, System software that controls the entire AI-equipped terminal; First software and a first AI model related to a function that requires availability among functions of the AI-equipped terminal; second software and a second AI model that can be updated by downloading from an external source; The second software and the second AI model are The system software, the first software, and the first AI model are virtually separated from each other; AI-equipped device.

2. An image to be subjected to the image recognition processing, which is captured by an imaging device, is input. The AI-equipped terminal according to claim 1.

3. an imaging device capable of capturing an image to be subjected to the image recognition processing is built in; The image captured by the imaging device is input. The AI-equipped terminal according to claim 1.

4. outputting the image to which frame information capable of identifying the image that has been subjected to the image recognition processing has been added; The AI-equipped terminal according to claim 1.

5. The image includes: The processing result of the image recognition processing is also added and output. The AI-equipped terminal according to claim 4.

6. An AI-equipped terminal capable of performing image recognition processing using artificial intelligence; A server that verifies the processing results of the AI-equipped terminal by image recognition processing using artificial intelligence and performs error correction; The AI-equipped terminal is System software that controls the entire AI-equipped terminal; First software and a first AI model related to a function that requires availability among functions of the AI-equipped terminal; second software and a second AI model that can be updated by downloading from an external source; The second software and the second AI model are The system software, the first software, and the first AI model are virtually separated from each other; AI system.

7. The server extracting a predetermined image from the image that has been subjected to the image recognition process by the AI-equipped terminal and performing a verification process; If there is an error in the processing result of the AI-equipped terminal, the error is corrected by interpolating data between the images that were the subject of the verification process. The AI system of claim 6.

8. The server a software library storing software for the AI-equipped terminal and the server; an AI model library in which AI models for the AI-equipped terminal and the server are stored; The AI-equipped terminal and the server are configured to be updatable by the software or the AI model. The AI system of claim 6.

9. The server Acquire from the AI-equipped terminal the image to which frame information capable of identifying the image that was the subject of the image recognition processing by the AI-equipped terminal is assigned, together with the processing result of the AI-equipped terminal; The AI system of claim 6.

10. An AI-equipped terminal capable of performing image recognition processing using artificial intelligence; A method executed by an AI system comprising: a server that verifies the processing result of the AI-equipped terminal by image recognition processing using artificial intelligence and performs error correction; The AI-equipped terminal is downloading software or AI models held on said server; A second software or a second AI model that is virtually separated from system software that controls the entire AI-equipped terminal and first software and a first AI model that are related to functions of the AI-equipped terminal that require availability, and that are related to the image recognition processing on the server, is updated to the software or the AI model downloaded from the server; method.

11. An AI-equipped terminal capable of performing image recognition processing using artificial intelligence; A program executed on a computer configured as an AI system including a server that verifies the processing results of the AI-equipped terminal and performs error correction through image recognition processing using artificial intelligence, downloading software or AI models held on said server; updating second software or a second AI model that is virtually separated from system software that controls the entire AI-equipped terminal and first software and a first AI model that are related to functions of the AI-equipped terminal that require availability, and that are related to the image recognition processing on the server, to the software or the AI model downloaded from the server; program.

Images