Information processing apparatus, information processing method, program, and recording medium

Abstract

To reduce a memory capacity of a database or data communication traffic between a sensing device and the database, while protecting privacy, in inference processing that uses an AI model based on data accumulated in the database from a sensing device.SOLUTION: An information processing system having a sensor includes: an input unit which inputs feature data from a database configured to store feature data obtained by a sensing device that performs feature extraction on sensing data obtained by the sensor using a backbone model which is an AI model configured to perform feature extraction on input data; a selection unit which selects a head model based on input information of a user, from among a plurality of candidates of a head model which is an AI model configured to perform inference processing using feature data as input data; and a control unit which controls the inference process to be performed using the feature data input by the input unit as input data, using the head model selected by the selection unit.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present technology relates to an information processing apparatus, an information processing method, a program, and a recording medium, and particularly relates to a technology for performing inference processing using an AI (Artificial Intelligence) model on sensing data obtained by a sensor.

Background Art

[0002] For example, there is a technology for performing inference processing using an AI (Artificial Intelligence) model on sensing data such as imaging image data obtained by a camera, and there is also a technology for performing various analyses on a sensing target based on the result of such inference processing. For example, it is conceivable to perform inference processing such as person detection processing using an AI model on imaging image data obtained by a camera installed in a store, and perform analyses such as counting the number of customers based on the inference result.

[0003] Note that the following Patent Document 1 can be cited as a related prior art. Patent Document 1 discloses a sensor-integrated inference device in which a signal processing unit that performs inference processing is mounted in an image sensor in which a pixel array unit is formed, as an inference device that performs inference processing on image data as target data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, consider the situation where output data from a sensing device having sensors is accumulated in a database, and various analyses regarding the sensing target are performed based on the accumulated data in the database. In this case, it is conceivable to adopt a configuration in which the database accumulates sensing data output by the sensing device, such as imaging image data by a camera, etc., and an information processing device on the subsequent stage performs inference processing by an AI model on the sensing data and analysis processing based on the inference result.

[0006] However, accumulating sensing data in a database as described above leads to an increase in the storage capacity of the database, which is not desirable. In particular, when the sensing data is image data (video data), it will lead to a significant increase in the storage capacity. Also, if the sensing data is accumulated in the database, the amount of communication data between the sensing device and the database also tends to increase, which is also not desirable in this regard.

[0007] Furthermore, considering that there is a possibility of data leakage between the sensing device and the database, accumulating sensing data in the database is not desirable from the perspective of privacy protection. In particular, when the sensing data is image data or sound data, it can be said that measures are necessary because it is easy to identify a person from the leaked data.

[0008] This technology has been made in view of the above circumstances, and when performing inference processing using an AI model based on data accumulated in a database from a sensing device, it aims to reduce the storage capacity of the database and the amount of communication data between the sensing device and the database while achieving privacy protection.

Means for Solving the Problem

[0009] The information processing apparatus according to the present technology has a sensor, and uses a backbone model, which is an AI model that extracts feature amounts of input data, to extract feature amounts of sensing data obtained by a sensing apparatus that extracts feature amounts of the sensing data by the sensor. The apparatus includes an input unit that inputs the feature amount data from a database in which the feature amount data is stored, a selection unit that selects the head model from among a plurality of candidates of the head model, which is an AI model that performs inference processing using the feature amount data as input data, based on input information of a user, and a control unit that controls so that inference processing using the feature amount data input by the input unit as input data is performed using the head model selected by the selection unit. The feature amount data extracted by the backbone model from the sensing data has a reduced data amount compared to the sensing data. Further, the feature amount data is obtained by extracting the feature amounts of the sensing target from the sensing data, and does not indicate the sensing target itself.

Brief Description of Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Modes for Carrying Out the Invention

[0011] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order. <1. First Embodiment> [1-1. Outline of System Configuration] [1-2. Configuration Example of Sensing Device] [1-3. Hardware Configuration Example of Information Processing Apparatus] [1-4. Inference Processing Method as the First Embodiment] [1-3. Processing Procedure] [1-4. Modification of the First Embodiment] <2. Second Embodiment> [2-1. Inference Processing Method as the Second Embodiment] [2-2. Another Example of the Second Embodiment] (2-2-1. The First Another Example) (2-2-2. The Second Another Example) <3. Third Embodiment> <4. Fourth Embodiment> <5. Modification Example> <6. Summary of Embodiments> <7. This Technology>

[0012] <1. First Embodiment> [1-1. Outline of System Configuration] FIG. 1 is a block diagram showing a schematic configuration example of an inference system as the first embodiment according to this technology. As shown in the figure, the inference system of the first embodiment includes an information processing apparatus 1, a sensing apparatus 3, a user terminal 4, and a database 5. The information processing apparatus 1, the user terminal 4, and the database 5 are each configured as a computer apparatus including a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). In this example, the information processing apparatus 1 is configured to be capable of performing data communication with the sensing apparatus 3, the user terminal 4, and the database 5 via a network 2 such as the Internet.

[0013] The sensing apparatus 3 is an apparatus configured to have a sensor that senses a physical phenomenon (that is, performs conversion of a physical quantity into an electric signal). In this example, an imaging apparatus is used as the sensing apparatus 3.

[0014] Here, in this specification, "imaging" is broadly defined as obtaining image data capturing a subject. The image data referred to here is a general term for data composed of a plurality of pixel data. As pixel data, it is not limited to data indicating the intensity of the amount of light received from the subject, but broadly includes, for example, information such as the distance to the subject, the polarization information of the subject, and temperature information. That is, the "image data" (imaging image data) obtained by "imaging" includes data as a gradation image indicating the intensity information of the amount of light received for each pixel, data as a distance image indicating the distance information to the subject for each pixel, or data as a polarization image indicating the polarization information of the incident light for each pixel, data as a thermal image indicating the temperature information for each pixel, and the like. Further, the "image data" also includes data as an event image obtained by an EVS (Event-based Vision Sensor) having a plurality of event detection pixels arranged two-dimensionally for detecting a change in the amount of light received as an event. This data as an event image can be paraphrased as image data indicating the presence or absence of an event for each pixel, and can be expressed as image data capturing the movement of the subject.

[0015] In the inference system, the user terminal 4 is a computer device assumed to be used by a user who is a recipient of a service using the inference system. Also, the information processing device 1 is a computer device assumed to be used by a service provider.

[0016] The database 5 is configured to have a storage device for storing data, and is used for storing the data obtained by the sensing device 3.

[0017] The inference system in this example is configured as a system for providing a service that performs inference processing using an AI (Artificial Intelligence) model on sensing data (imaging image data in this example) by the sensing device 3, generates analysis information indicating an analysis result for the sensing target based on the inference result, and presents it to the user.

[0018] Here, when assuming a service of presenting analysis information of the sensing target based on the inference result to the user as described above, as the use of the sensing device 3, uses of various monitoring cameras can be considered. For example, monitoring cameras for indoor areas such as stores, offices, and houses, monitoring cameras for outdoor areas such as parking lots and streets (including traffic monitoring cameras, etc.), monitoring cameras for production lines in FA (Factory Automation) and IA (Industrial Automation), and monitoring cameras for monitoring inside and outside vehicles, etc. can be cited.

[0019] For example, in the case of the use of a monitoring camera in a store, it is conceivable to arrange a plurality of sensing devices 3 at predetermined positions inside the store so that the user can confirm the customer layer (such as gender and age group) of the customers visiting the store and the behavior (flow line) inside the store. In that case, as the above-described analysis information, it is conceivable to generate information on the customer layer of these customers visiting the store, information on the flow line inside the store, and information on the congestion state at the checkout register (for example, waiting time information at the checkout register), etc. Alternatively, in the case of the use of a traffic monitoring camera, each sensing device 3 is arranged at each position near the road so that the user can recognize information such as the number (license plate number), color of the car, and vehicle type of the passing vehicles. In that case, as the above-described analysis information, it is conceivable to generate information such as these numbers, colors of the cars, and vehicle types.

[0020] Also, when using a traffic monitoring camera in a parking lot, cameras are arranged so that each parked vehicle can be monitored, and it is monitored whether there are any suspicious persons performing suspicious actions around each vehicle. If there is a suspicious person, it is conceivable to notify the fact that there is a suspicious person and the attributes of the suspicious person (such as gender, age group, clothing, etc.). Furthermore, it is also conceivable to monitor the vacant spaces in the street or parking lot and notify the user of the locations where the vehicle can be parked.

[0021] In FIG. 1, an example is shown in which the inference system includes a plurality of sensing devices 3, but the number of sensing devices 3 may be at least one or more. Also, in FIG. 1, the number of user terminals 4 in the inference system is set to one, but there may be a plurality of user terminals 4. That is, it is also assumed that the number of users who are the recipients of the services provided by the inference system is plural.

[0022] [1-2. Configuration Example of Sensing Device] FIG. 2 is a block diagram showing a configuration example of the sensing device 3. As shown in the figure, the sensing device 3 includes an image sensor 30, an imaging optical system 31, an optical system drive unit 32, a camera control unit 33, a memory unit 34, a communication unit 35, and a sensor unit 36. The image sensor 30, the camera control unit 33, the memory unit 34, the communication unit 35, and the sensor unit 36 are connected via a bus 37 and are capable of performing data communication with each other.

[0023] In this example, the image sensor 30 is configured as a tone image sensor for obtaining the tone image described above. That is, the sensing device 3 in this example is configured as a camera device that obtains a tone image as a captured image.

[0024] The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, and a diaphragm (iris) mechanism. The imaging optical system 31 guides light (incident light) from the subject and focuses it on the light receiving surface of the image sensor 30.

[0025] The optical system drive unit 32 comprehensively shows the drive units of the zoom lens, the focus lens, and the diaphragm mechanism included in the imaging optical system 31. Specifically, the optical system drive unit 32 has actuators for driving these zoom lens, focus lens, and diaphragm mechanism, respectively, and a drive circuit for the actuators.

[0026] The camera control unit 33 is configured to include a microcomputer having, for example, a CPU, a ROM, and a RAM. The CPU executes various processes according to a program stored in the ROM or a program loaded into the RAM, thereby performing overall control of the sensing device 3.

[0027] In addition, the camera control unit 33 issues drive instructions for a zoom lens, a focus lens, a diaphragm mechanism, etc. to the optical system drive unit 32. The optical system drive unit 32 causes the movement of the focus lens and the zoom lens, the opening and closing of the diaphragm blades of the diaphragm mechanism, etc. to be executed according to these drive instructions.

[0028] In addition, the camera control unit 33 controls the writing and reading of various data to and from the memory unit 34. The memory unit 34 is a non-volatile storage device such as an HDD (Hard Disc Drive) or a flash memory device, and is used for storing data used when the camera control unit 33 executes various processes. Also, the memory unit 34 can be used as a storage destination (recording destination) for the image data output from the image sensor 30.

[0029] The camera control unit 33 performs various data communications with an external device via the communication unit 35. The communication unit 35 in this example is capable of communicating via the network 2 shown in FIG. 1, and is configured to be capable of performing data communication with an external device connected to the network 2, particularly in this example, at least with the database 5 and the information processing device 1.

[0030] The sensor unit 36 comprehensively represents sensors other than the image sensor 30 included in the sensing device 3. Examples of the sensors included in the sensor unit 36 include a GNSS (Global Navigation Satellite System) sensor and an altitude sensor for detecting the position and altitude of the sensing device 3, a temperature sensor for detecting the ambient temperature, and motion sensors such as an acceleration sensor and an angular velocity sensor for detecting the motion of the sensing device 3. Furthermore, an illuminance sensor for detecting the illuminance of external light with respect to the sensing device 3, a sensor such as a microphone for picking up the ambient sound of the sensing device 3, etc. can also be mentioned.

[0031] The image sensor 30 is configured as a solid-state imaging device such as a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type, and includes an imaging unit 41, an image signal processing unit 42, an in-sensor control unit 43, an AI processing unit 44, a memory unit 45, a computer vision processing unit 46, and a communication interface (I / F) 47 as shown in the figure, and each can perform data communication with each other via a bus 48.

[0032] The imaging unit 41 includes a pixel array unit in which pixels having photoelectric conversion elements such as photodiodes are two-dimensionally arranged, and a readout circuit that reads out the electrical signals obtained by photoelectric conversion from each pixel included in the pixel array unit. In this readout circuit, for the electrical signals obtained by photoelectric conversion, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. are executed, and further A / D (Analog / Digital) conversion processing is performed.

[0033] The image signal processing unit 42 performs preprocessing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, etc. on the captured image signal as digital data after A / D conversion processing. In the preprocessing, operations such as clamp processing for clamping the black levels of R (red), G (green), and B (blue) in the captured image signal to a predetermined level, and correction processing between the color channels of R, G, and B are performed. Also, in the preprocessing, adjustment processing related to brightness such as gamma correction processing, and adjustment processing related to color such as white balance adjustment processing can be performed. In the synchronization processing, color separation processing is performed so that the image data for each pixel has all color components of R, G, and B. For example, in the case of an image sensor using a Bayer array color filter, demosaicing processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion processing, resolution conversion processing is executed on the image data subjected to various signal processes. In the codec processing, for the image data subjected to the above various processes, for example, encoding processes for recording and communication, and file generation are performed. In the codec processing, as the file format of the moving image, for example, file generation can be performed in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264. Also, it is conceivable to perform file generation in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) as the still image file.

[0034] The in-sensor control unit 43 includes, for example, a microcomputer configured to have a CPU, a ROM, a RAM, etc., and comprehensively controls the operation of the image sensor 30. For example, the in-sensor control unit 43 gives an instruction to the imaging unit 41 to perform execution control of the imaging operation. Also, execution control of the processing is performed on the image signal processing unit 42.

[0035] The AI processing unit 44 is configured to include a programmable arithmetic processing device such as a CPU, an FPGA (Field Programmable Gate Array), or a DSP (Digital Signal Processor), and performs processing (AI processing) using an AI model on the captured image. The AI processing unit 44 in this example is configured to be able to switch the AI model to be used. Details of the AI processing performed by the AI processing unit 44 will be described later again.

[0036] The memory unit 45 is composed of a volatile memory and is used to hold (temporarily store) data necessary for performing AI processing by the AI processing unit 44. Specifically, the memory unit 45 is used to store AI model data, which is necessary parameter data for constructing an AI model in the AI processing unit 44. When this AI model data is for an AI model having a neural network such as a CNN (Convolutional Neural Network), it corresponds to parameters indicating the structure of the neural network and parameters as filter coefficients used in convolution processing and the like. In this example, the memory unit 45 is also used to hold the captured image data that is input data to the AI processing unit 44.

[0037] The computer vision processing unit 46 performs rule-based image processing as image processing on the captured image data. Examples of the rule-based image processing here include super-resolution processing and the like.

[0038] The communication interface 47 is an interface that communicates with each unit connected via the bus 37, such as the camera control unit 33 and the memory unit 34 outside the image sensor 30. For example, based on the control of the in-sensor control unit 43, the communication interface 47 communicates to acquire from the outside AI model data and the like for realizing the AI model in the AI processing unit 44. Also, it is possible to output the result information of the AI processing by the AI processing unit 44 and the like to the outside of the image sensor 30 via the communication interface 47.

[0039] [1-3. Hardware Configuration Example of Information Processing Apparatus] FIG. 3 is a block diagram showing a hardware configuration example of the information processing apparatus 1. Note that the computer devices as the user terminal 4 and the database 5 shown in FIG. 1 may also adopt the same hardware configuration as that shown in FIG. 3 (excluding the AI processing unit 14).

[0040] As shown in the figure, the information processing apparatus 1 includes a CPU 11. The CPU 11 executes various processes according to a program stored in the ROM 12 or a program loaded from the storage unit 19 to the RAM 13. The RAM 13 also appropriately stores data and the like necessary for the CPU 11 to execute various processes.

[0041] The AI processing unit 14 is configured to have a programmable arithmetic processing device such as a CPU, an FPGA, or a DSP, similar to the AI processing unit 44 described above, and performs processing (AI processing) using an AI model on the captured image. This AI processing unit 14 is also configured to be able to switch the AI model to be used, similar to the AI processing unit 44. Note that the details of the AI processing performed by the AI processing unit 14 will be described again later.

[0042] The CPU 11, the ROM 12, the RAM 13, and the AI processing unit 14 are interconnected via a bus 23. An input / output interface (I / F) 15 is also connected to this bus 23.

[0043] An input unit 16 composed of an operator and an operation device is connected to the input / output interface 15. For example, as the input unit 16, various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, and a remote controller are assumed. The input unit 16 detects the user's operation, and the signal corresponding to the input operation is interpreted by the CPU 11.

[0044] Also, a display unit 17 composed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, etc., and an audio output unit 18 composed of a speaker, etc. are connected to the input / output interface 15 integrally or separately. The display unit 17 is used for displaying various kinds of information, and is composed of, for example, a display device provided on the housing of the computer device, a separate display device connected to the computer device, etc.

[0045] The display unit 17 executes the display of images for various image processes, moving images of processing targets, etc. on the display screen based on the instructions of the CPU 11. Also, the display unit 17 performs the display as a GUI (Graphical User Interface), such as various operation menus, icons, messages, etc., based on the instructions of the CPU 11.

[0046] The input / output interface 15 may also be connected to a storage unit 19 composed of an HDD, a solid-state memory, etc., and a communication unit 20 composed of a modem, etc.

[0047] The communication unit 20 performs communication processing via a transmission path such as the Internet, wired / wireless communication with various devices, bus communication, etc.

[0048] The input / output interface 15 is also connected to a drive 21 as needed, and a removable recording medium 22 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.

[0049] The drive 21 can read data files such as programs used for each process from the removable recording medium 22. The read data files can be stored in the storage unit 19, or images and sounds included in the data files can be output by the display unit 17 and the audio output unit 18. Also, computer programs and the like read from the removable recording medium 22 are installed in the storage unit 19 as necessary.

[0050] In a computer device having the above hardware configuration, for example, software for the processing of the present embodiment can be installed via network communication by the communication unit 20 or via the removable recording medium 22. Alternatively, the software may be stored in advance in the ROM 12, the storage unit 19, or the like. When the CPU 11 performs a processing operation based on various programs, necessary information processing and communication processing as the information processing apparatus 1 are executed.

[0051] Note that the information processing apparatus 1 is not limited to being configured by a single computer device as shown in FIG. 3, and a plurality of computer devices may be configured as a system. The plurality of computer devices may be systematized by a LAN (Local Area Network) or the like, or may be arranged remotely by a VPN (Virtual Private Network) or the like using the Internet or the like. The plurality of computer devices may include computer devices as a server group (cloud) that can be used by a cloud computing service.

[0052] [1-4. Inference Processing Method as the First Embodiment] First, the basic concept of the inference processing method as the embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 is an explanatory diagram of the functional division of the AI model assumed in the embodiment, and FIG. 4A schematically shows the network structure of the AI model using a DNN (Deep Neural Network). Generally, in an AI model, when the input layer side in the network is regarded as the front side and the output layer side is regarded as the rear side, the front part up to a predetermined intermediate layer is called the backbone part, and it functions as a part for extracting the feature amounts of the sensing target from the input data. Also, the part after the backbone part in the network functions as a part for performing processing to obtain an inference result using the extracted feature amounts as input data, and here, this part will be called the head part.

[0053] In the present embodiment, regarding the AI model that performs a series of processing from feature amount extraction to obtaining an inference result, as shown in FIG. 4B, the backbone part that is the front part of the network and the head part that is the rear part thereof are divided and handled. Hereinafter, the AI model that performs processing as the backbone part and the AI model that performs processing as the head part will be denoted as "backbone model BB" and "head model HD", respectively, as shown in the figure.

[0054] Note that the backbone part and the head part may each be composed of a plurality of models. For example, the backbone model BB may have a neck model NC that performs feature amount extraction and feature amount integration, and the head model HD may use the feature amounts output from the neck model NC as input data.

[0055] Also, in the following description, the data indicating the feature amounts extracted from the input data by the backbone model BB will be denoted as "feature amount data".

[0056] FIG. 5 is an explanatory diagram of the basic concept of the inference processing method as an embodiment. As shown in the figure, in the embodiment, the feature amount extraction is performed using the backbone model BB on the sensing device 3 side, and the information processing device 1 executes inference processing using the head model HD with the feature amount data extracted by the backbone model BB as input data.

[0057] At this time, it is conceivable that the backbone model BB and the head model HD learn an AI model including the entire network from the input layer to the output layer as shown in FIG. 4A above, and obtain them by dividing the AI model after machine learning. In this example, the AI processing is AI processing for image data, and for example, it is conceivable to perform object detection processing, object recognition processing, etc. An AI model for realizing a predetermined inference task such as these object detections and object recognitions is machine-learned, and the learned AI model is divided at a predetermined intermediate layer position to obtain a backbone model BB and a head model HD.

[0058] Here, depending on the type of inference task, there may be cases where the backbone model BB can be shared among different head models HD. For example, in such a case, it is also conceivable to separately learn the backbone model BB and the head model HD.

[0059] Based on the above premises, with reference to FIG. 6, an inference processing method as the first embodiment will be described. First, in the present embodiment, the feature amount data extracted by the sensing device 3 using the backbone model BB is temporarily stored in the database 5. As shown in the figure, in this example, the feature amount data obtained by each of the plurality of sensing devices 3 is stored in the database 5. In this example, the sensing device 3 as an imaging device performs feature amount extraction using the backbone model BB for each frame of the captured image, and the feature amount data for each frame is stored in the database from each sensing device 3.

[0060] Also, in this example, the sensing device 3 adds attached information to the feature amount data. The attached information is to be added to the feature amount data of each frame, and examples thereof include a time stamp (information representing the sensing time: the time here includes the concept of date) and information on the sensing location. Note that the information on the sensing location can be acquired by the GNSS sensor included in the sensor unit 36 shown in FIG. 2. In addition, the attached information may include detection information obtained by various sensors included in the sensor unit 36 such as the illuminance sensor and the temperature sensor described above, and information indicating the weather at the time of sensing (for example, information acquired by the sensing device 3 from the Internet). Furthermore, as the attached information, for example, parameter information of image signal processing performed by the image signal processing unit 42 (for example, parameters such as gamma correction and white balance adjustment), parameter information of signal processing performed on the captured image, drive information (accumulation time) of the sensor, etc. may also be included.

[0061] In the sensing device 3, for example, the camera control unit 33 adds the above-mentioned attached information to the feature amount data obtained by the image sensor 30, and transmits the feature amount data to which the attached information is added to the database 5 via the communication unit 35 for storage.

[0062] Hereinafter, the set of the feature amount data and the attached information stored in the database 5 from each sensing device 3 as described above is referred to as "accumulated information 5a".

[0063] The information processing device 1 performs inference processing using the head model HD on the feature amount data stored in the database 5 as the accumulated information 5a as described above. Specifically, the information processing device 1 includes an input unit F1 that inputs feature amount data from the database 5, a selection unit F2 that selects a head model HD based on the user's input information from among a plurality of candidates of the head model HD, and a control unit F3 that controls the AI processing unit 14 to perform inference processing using the feature amount data input by the input unit F1 as input data using the head model HD selected by the selection unit F2. In this example, these input unit F1, selection unit F2, and control unit F3 are functional units realized by software processing of the CPU 11 in the information processing device 1.

[0064] The information processing apparatus 1 in this example includes a model storage unit F4. A plurality of head models HD that are selection candidates by the selection unit F2 are stored in this model storage unit F4. The model storage unit F4 is a functional unit realized by a storage device included in the information processing apparatus 1, such as the storage unit 19 or the like.

[0065] In this example, the input information of the user to the selection unit F2 is given from the user terminal 4 as shown in the figure. In this example, as this input information of the user, information indicating the target task to be realized using the inference process is input, and the selection unit F2 selects a head model HD capable of executing the inference task specified from this target task. In this example, the input information of the user is information representing the target task in a language form. Specifically, the input information of the user is input in the form of text data. Note that the user input information in a language form is not limited to that by text data. For example, it may be in the form of voice input data using a microphone. In that case, the input voice data may be converted into text data and used for the selection by the selection unit F2.

[0066] In this example, it is assumed that the input information of the user indicating the target task is input by text such as "want to know the traffic volume of people at ○○ intersection today" or "want to know the number of visitors to store A yesterday morning". That is, it is assumed that the input information includes designation information for designating time or place.

[0067] Correspondingly, the selection unit F2 in this example performs a process of selecting, as the feature amount data input by the input unit F1, the feature amount data corresponding to the time or place designated by the above-mentioned designation information from among the feature amount data stored in the database 5. Thereby, when the feature amount data about the sensing data at various times and places is accumulated in the database 5, it is possible to plan to perform an inference process for the sensing data at the time and place desired by the user.

[0068] Also, in this example, the selection unit F2 selects the head model HD and feature data based on the input information using an LLM (Large Language Model). That is, using the language format data indicating the target task input by the user as a prompt, it searches for the head model HD and feature data using the LLM, and selects the head model HD and feature data necessary for realizing the target task.

[0069] In order to cope with such a search using an LLM, in the model storage unit F4, search tag information is attached to each head model HD. Specifically, as this tag information, at least information indicating the type of inference task of the inference process executed by the head model HD (hereinafter referred to as "inference task identification information") is attached.

[0070] For example, for the input of the above-mentioned target task of "wanting to know the traffic volume of people at ○○ intersection today", the selection unit F2 selects from the model storage unit F4 a head model HD to which inference task identification information indicating an object detection process for "people" is added, and also selects feature data to which location information corresponding to "○○ intersection" is added and a time stamp corresponding to "today" is added from the database 5.

[0071] Regarding the selection process by the selection unit F2, by using an LLM, it becomes possible to accurately search for the head model HD and feature data that match the task conditions specified by the user in a language format. Therefore, the accuracy of appropriately executing the target task desired by the user can be increased.

[0072] Also, the information processing apparatus 1 has functions as an analysis unit F5 and a presentation processing unit F6. In this example, the functions of these analysis unit F5 and presentation processing unit F6 are also functions realized by software processing of the CPU 11.

[0073] The analysis unit F5 inputs the inference result information by the AI processing unit 14 (that is, information indicating the inference result corresponding to the target task specified by the user), performs analysis processing using the input inference result information, and generates analysis information of the sensing target. The analysis unit F5 performs analysis processing according to the target task specified by the user with the input information. For example, when the specified target task is "want to know the traffic volume of people at a certain intersection today", the analysis unit F5 performs processing such as counting the number of people using the person detection result information obtained by the AI processing unit 14, and generates information indicating the traffic volume of people at the corresponding location and time as the analysis information.

[0074] The presentation processing unit F6 performs processing to present the analysis information obtained by the analysis unit F5 to the user as visualization information. FIG. 7 illustrates visualization information of the analysis information. Specifically, it is information obtained by visualizing the traffic volume of people in a certain period at a certain location for each predetermined time zone. The presentation processing unit F6 generates, for example, image information showing the analysis result as shown in FIG. 7 as visualization information of the analysis information, and performs processing to present the generated visualization information to the user via the user terminal 4. That is, the image information as the visualization information is transmitted to the user terminal 4 and displayed on the display screen of the user terminal 4.

[0075] In the above, an example in which the presentation processing unit F6 generates visualization information of the analysis information is given, but a configuration in which the analysis unit F5 performs up to the generation of the visualization information is also conceivable.

[0076] Also, in the above, an example in which candidates for the head model HD are stored in the model storage unit F4 in the information processing apparatus 1 has been described, but candidates for the head model HD may be stored in a storage device outside the information processing apparatus 1, such as a database 5.

[0077] [1-3. Processing procedure] FIG. 8 is a flowchart showing an example of a processing procedure for realizing the inference processing method as the first embodiment described above. In this example, the process shown in FIG. 8 is executed by the CPU 11 in the information processing apparatus 1 based on a program stored in, for example, the ROM or the storage unit 19.

[0078] Here, when the process shown in FIG. 8 is executed, it is assumed that the feature amount data extracted by the backbone model BB from the sensing device 3 and its attached information have already been accumulated in the database 5. For the sensing device 3, the backbone model BB necessary for realizing the target task designated by the user's input information is deployed. Specifically, this backbone model BB is selected by the information processing apparatus 1 (CPU 11) from a plurality of candidates based on the user's input information, and the selected model is deployed to the target sensing device 3. For confirmation, "deployment" means transmitting an AI model to a target device for installation. Specifically, it performs the transmission of the above-described AI model data and the setting instruction of the AI model based on the AI model data.

[0079] In FIG. 8, in step S101, the CPU 11 performs a search process for the head model HD and the feature amount data based on the user input information. Specifically, in this example, a search process for the head model HD and the feature amount data using the LLM is performed with the information indicating the target task input by the user in a language format as a prompt.

[0080] In step S102 following step S101, the CPU 11 sets the head model HD selected based on the search result in the AI processing unit 14. Specifically, according to the AI model data for realizing the head model HD, the corresponding AI model is set in the AI processing unit 14.

[0081] In step S103 following step S102, the CPU 11 inputs the feature amount data selected based on the search result. That is, based on the search result of step S101, the corresponding feature amount data is selected from the accumulated information 5a in the database 5, and the selected feature amount data is input.

[0082] In step S104 following step S103, the CPU 11 causes the AI processing unit 14 to execute an inference process using the input feature amount data as input data.

[0083] In step S105 following step S104, the CPU 11 executes an analysis process based on the inference result, and in subsequent step S106, performs a process of presenting visualization information of the analysis information to the user. Thereby, information indicating the result of the target task specified by the user can be presented to the user as visualization information.

[0084] In response to executing the process of step S106, the CPU 11 finishes a series of processes shown in FIG. 8.

[0085] [1-4. Modification Example of the First Embodiment] Here, in the above description, it was premised that the number of head models HD required for realizing the target task specified by the user is one, but as the target task, the number of head models HD required for its realization may be plural. For example, in the case of the target task of "wanting to know the traffic volume of people and cars today at ○○ intersection", as illustrated in FIG. 9, in the information processing apparatus 1, using the head model HD for person detection ( "HDh" in the figure) and the head model HD for vehicle detection ( "HDc" in the figure), an inference process should be performed using common feature amount data (feature amount data selected by the selection unit F2) as input data.

[0086] Therefore, when the number of head models HD required for realizing the target task specified from the input information of the user is plural, the selection unit F2 selects each head model HD (that is, a plurality of head models HD with different inference tasks) from the model storage unit F4, and the control unit F3 causes each head model HD selected by the selection unit F2 to execute an inference process using the same feature amount data input by the input unit F1 as input data.

[0087] As a result, for the same sensing target, inference processing for different tasks can be performed simultaneously in parallel. In addition, when performing inference processing for different tasks, only a single model needs to be used as the backbone model BB, and it is not necessary to perform the process of extracting features using the backbone model BB for each task on the sensing device 3 side. Therefore, the processing load on the sensing device 3 can be reduced.

[0088] <2. Second Embodiment> [2-1. Inference Processing Method as the Second Embodiment] Subsequently, the second embodiment will be described. As shown in FIG. 10, in the second embodiment, the head model HD is deployed to the sensing device 3, and the sensing device 3 is caused to execute inference processing for a predetermined task. The information processing device of the second embodiment in which the head model HD is deployed to the sensing device 3 in this way is denoted as the information processing device 1A. The hardware configuration of the information processing device 1A may be the same as that of the information processing device 1, and redundant explanations will be avoided.

[0089] In the following description, the same parts as those already described will be denoted by the same reference numerals and the description will be omitted.

[0090] The second embodiment is intended to be applicable, for example, when the target task specified by the user's input information requires inference processing for data during sensing rather than data stored in the database 5. In this case, examples of the target task may include "want to know the current traffic volume of people at the ○○ intersection".

[0091] In this case, the selection unit F2 in the information processing device 1A selects the corresponding sensing device 3 and the corresponding head model HD based on the target task specified by the user. Then, the selected head model HD is deployed to the selected sensing device 3 to execute inference processing. At this time, the information on the inference result obtained by the sensing device 3 may be once stored in the database 5 and then transferred to the information processing device 1A, or may be transmitted from the sensing device 3 to the information processing device 1A without passing through the database 5. In this case, regarding the analysis process and presentation process based on the inference result information, since they are the same as the processes of the analysis unit F5 and the presentation processing unit F6 described in the first embodiment, duplicate explanations will be avoided.

[0092] Also, in the second embodiment, in response to a case where the target task specified by the user's input information requires a plurality of types of inference processes for the data being sensed, when the selection unit F2 selects a plurality of head models HD, as shown in FIG. 11, those head models HD (HD1, HD2 in the figure) are deployed to the sensing device 3, and an instruction to switch the head model HD to be used among the deployed head models HD is given to the sensing device 3.

[0093] Thereby, as an inference process for the data being sensed, it becomes possible to execute the inference processes by a plurality of head models HD in a time-sharing manner on one sensing device 3, and it is not necessary to deploy each head model HD to a plurality of sensing devices 3 for execution, so that the number of sensing devices can be reduced.

[0094] The information processing device 1A of the second embodiment is different from the information processing device 1 in that, as shown in FIG. 12, it includes a CPU 11A having a control unit F3A instead of the control unit F3. The control unit F3A controls so that the head model HD selected by the selection unit F2 is deployed to the sensing device 3. In particular, when the selection unit F2 selects a plurality of head models HD based on the user's input information, the control unit F3A controls so that those plurality of head models HD are deployed to the target sensing device 3, and an instruction to switch the head model HD to be used among the deployed head models HD is given to the sensing device 3.

[0095] In the above description, as an example where the selection unit F2 selects a plurality of head models HD, a case where there are a plurality of head models HD required for realizing the target task specified by a single user has been exemplified. However, the selection unit F2 may also select a plurality of head models HD for realizing the target tasks of respective users based on input information from a plurality of users who request execution of different target tasks. In response to such a case, when the control unit F3A performs the above processing, it becomes possible to execute the inference processing for the target tasks of respective users in a time-division manner in one sensing device 3. Therefore, when realizing the target tasks of a plurality of users respectively, it is not necessary to prepare a sensing device 3 for each user, and the number of sensing devices can be reduced.

[0096] Note that, depending on the deployment of a plurality of head models HD to the sensing device 3 as described above, it is also possible to execute the inference processing for a plurality of real-time sensing data in the sensing device 3 simultaneously and in parallel. In this case, the sensing device 3 is provided with a plurality of AI processing units 44 for operating a plurality of head models HD simultaneously and in parallel. In this case, the applications are not limited to executing a plurality of tasks simultaneously and in parallel. For example, when the number of classes that can be identified by only one head model HD is insufficient, applications such as increasing the number of head models HD to increase the number of classes that can be identified are also conceivable.

[0097] Also, in the second embodiment, as shown in FIG. 13, the information processing device 1A not only deploys a plurality of head models HD but also deploys a plurality of backbone models BB (BB1, BB2 in the figure), and it is also conceivable to give an instruction to switch the backbone model BB. Thereby, it becomes possible to cope with a case where the backbone model BB corresponding to each head model HD is different.

[0098] At this time, the switching of the backbone model BB may include switching to a backbone model BB having a different number of layers. This makes it possible to handle cases where the number of layers of the backbone model BB that can be supported varies for each head model HD.

[0099] For confirmation, it should be noted that, also in the case where the execution of the inference process targeting the feature amount data in the storage information 5a is instructed by the user, as in the control unit F3 of the first embodiment, when the inference process using the feature amount data input by the input unit F1 as input data is performed using the head model HD selected by the selection unit F2, control is performed. Similarly, for the control units (control units F3B to F3E) of each embodiment described hereinafter, when the execution of the inference process targeting the feature amount data in the storage information 5a is instructed by the user, control is performed so that the inference process using the feature amount data input by the input unit F1 as input data is performed using the head model HD selected by the selection unit F2.

[0100] [2-2. Another Example of the Second Embodiment] (2-2-1. The First Another Example) An information processing apparatus as another example of the second embodiment will be described. Here, two examples, the first another example and the second another example, will be described, and the information processing apparatus as the first another example is denoted as the information processing apparatus 1B, and the information processing apparatus as the second another example is denoted as the information processing apparatus 1C.

[0101] As shown in FIG. 14, the information processing apparatus 1B as the first another example gives an instruction to transfer the deployed head model HD to another sensing device 3 to the sensing device 3 where the head model HD is deployed.

[0102] The information processing apparatus 1B is different from the information processing apparatus 1 in that it includes a CPU 11B having a control unit F3A instead of the control unit F3 as shown in FIG. 15. The control unit F3B controls so that the head model HD selected by the selection unit F2 is deployed to the target sensing device 3, and gives an instruction to the deployed sensing device 3 to transfer the deployed head model HD to another sensing device 3.

[0103] As a result, when it is required to perform inference processing using the head model HD selected by the selection unit F2 not only in a single sensing device 3 but also in another sensing device 3, it becomes unnecessary to perform the deployment process of the head model HD to another sensing device 3.

[0104] Here, the transfer function of the head model HD as the first alternative example as described above can be suitably applied, for example, to the tracking of specific objects such as criminals and stolen vehicles. In that case, the selection unit F2 selects a head model HD that performs object detection processing targeting a specific object. Then, the control unit F3B gives the following instruction as the above transfer instruction to the deployed sensing device 3. That is, an instruction indicating that the deployed head model HD is to be transferred to another sensing device 3 on the condition that an inference result that a specific object has been detected is obtained by inference processing using the deployed head model HD is given.

[0105] As a result, in the sensing device 3 in which the head model HD is deployed, it is possible to ensure that the head model HD for detecting the specific object is transferred to another surrounding sensing device 3 in response to the detection of a specific object such as a criminal or a stolen vehicle, and it is possible to enable the tracking of the specific object.

[0106] At this time, if the positions of the respective sensing devices 3 are known, the deployed sensing device 3 can also identify another sensing device 3 located in the moving direction of the specific object (such as by performing estimation processing of the moving direction of the detected specific object) and transfer the head model HD to that sensing device 3.

[0107] (2-2-2. Second alternative example) The second alternative example is not to issue a transfer instruction for the deployed head model HD, but to issue an instruction to send the feature amount data extracted by the backbone model BB to another sensing device 3. Specifically, as shown in FIG. 16, the information processing apparatus 1C as the second alternative example causes a plurality of head models HD selected based on the input information of the user to be respectively deployed to different sensing devices 3, and issues an instruction to one of the deployment destination sensing devices 3 to send the feature amount data extracted by the one sensing device 3 to the other deployment destination sensing devices 3.

[0108] Thereby, when a plurality of types of inference processes should be performed on one piece of sensing data, these inference processes can be realized as distributed processing using a plurality of sensing devices 3.

[0109] The information processing apparatus 1C is different from the information processing apparatus 1 in that it includes a CPU 11C having a control unit F3C instead of the control unit F3 as shown in FIG. 17. The control unit F3C controls so that a plurality of head models HD selected by the selection unit F2 are respectively deployed to different sensing devices 3. In this case, the selection unit F2 selects a plurality of head models HD in response to a case where a target task that requires execution of a plurality of different inference processes is specified for the sensing data at a single location based on the input information of a single user. Then, the control unit F3C performs deployment control of the plurality of head models HD selected by the selection unit F2 as described above, and issues an instruction to one of the deployment destination sensing devices 3 to send the feature amount data extracted by the backbone model BB of the one sensing device 3 to the other deployment destination sensing devices 3.

[0110] <3. Third Embodiment> The third embodiment relates to the relearning of the backbone model BB. Referring to FIG. 18, the information processing apparatus 1D as the third embodiment will be described. The information processing apparatus 1D is different from the information processing apparatus 1 in that it includes a control unit F3D instead of the control unit F3 and also includes a relearning processing unit F7.

[0111] The control unit F3D performs execution control of relearning for the backbone model BB used by the sensing device 3 in response to the establishment of a predetermined trigger condition, and controls so that the backbone model BB (hereinafter, denoted as "backbone model BB'" as shown in the figure) obtained by the relearning is deployed to the sensing device 3. In this example, the "predetermined trigger condition" as the execution condition of relearning is a condition based on an evaluation of the accuracy of the inference process performed using the head model HD. Specifically, the control unit F3D of this example causes the relearning processing unit F7 to perform relearning on the backbone model BB used by the sensing device 3 on the condition that an error has been detected in the inference result of the inference process performed using the head model HD with the feature amount data accumulated in the database 5 from the sensing device 3 as input data. Whether there is an error in the inference result can be determined by the control unit F3D itself performing the error detection process. For example, the error detection process can be realized as a determination process of whether an impossible inference result has been obtained. Specifically, for example, in the case of object detection processing for a specific object, when there is a premise that at least one specific object is detected in the sensing environment, it is conceivable to perform the determination of whether the number of detected specific objects is 0 as the error detection process. Or, when the same area is sensed by the sensing devices 3 at different angles and the inference process for the same task is executed for the sensing data by each sensing device 3, it is also conceivable to perform the determination of whether one sensing device 3 can detect the objects detected by the other plurality of sensing devices 3 as the error detection process. Furthermore, there may be a case where an error is pointed out by feedback of analysis information to the user, and in that case, it is also conceivable to perform the determination process of whether the error has been pointed out as the error detection process.

[0112] In the above, as an "evaluation of the accuracy of the inference process", an example was given in which the evaluation was made based on the presence or absence of an error in the inference result. However, as the "evaluation of the accuracy of the inference process", it is also conceivable to use an evaluation based on a score (likelihood) of the inference result. For example, in the case of performing object recognition processing, it is also conceivable to perform a determination as to whether or not the average score value in the time direction for a certain recognition target object is below a threshold value, or a determination as to whether or not the average score value of a plurality of recognition target objects is below a threshold value, as a determination of whether or not the trigger condition is satisfied.

[0113] FIG. 19 is a flowchart showing an example of a processing procedure for realizing the inference processing method as the third embodiment described above. The processing shown in FIG. 19 is executed by the CPU 11 (denoted as "CPU 11D" for distinction) in the information processing apparatus 1D based on a program stored in the ROM or the storage unit 19 in the information processing apparatus 1D.

[0114] First, in step S201, the CPU 11D waits until the retraining condition of the backbone model BB is satisfied. Specifically, in this example, the result of the inference process by the head model HD performed using the feature amount data input from the target sensing device 3 via the database 5 as input data is input, and the above-described error detection process is executed.

[0115] In step S201, when an error in the inference result is detected by the error detection process and it is determined that the retraining condition is satisfied, the CPU 11D proceeds to step S202 and gives an instruction to execute retraining of the backbone model BB to the retraining processing unit F7.

[0116] Then, in step S203 following step S202, the CPU 11D performs a deployment process of the re - learned backbone model BB' to the sensing device that is the source of the backbone model BB'. That is, for the target sensing device 3, control is performed so that the backbone model BB' obtained by the re - learning process in the re - learning processing unit F7 is deployed. At this time, the CPU 11D also gives an instruction to the target sensing device 3 to switch the backbone model to be used to the backbone model BB'.

[0117] In response to executing the process of step S203, the CPU 11D finishes a series of processes shown in FIG. 19.

[0118] In the above, an example in which the re - learning processing unit F7 is provided in the information processing device 1D has been given. However, the re - learning process may be performed by a device different from the information processing device 1D.

[0119] Also, in the above, the case where the "predetermined trigger condition" as the execution condition of re - learning is the "condition based on the evaluation of the accuracy of the inference process" has been exemplified. However, the "predetermined trigger condition" is not limited to this. For example, when it becomes necessary to increase the objects to be detected or recognized due to the specification of a new target task from the user, it is also conceivable to execute the re - learning of the backbone model BB.

[0120] <4. Fourth Embodiment> The fourth embodiment relates to the optimization of the backbone model BB according to the sensing environment. Referring to FIG. 20, the information processing device 1E as the fourth embodiment will be described. As shown in the figure, the information processing device 1E is different from the information processing device 1 in that it includes a control unit F3E instead of the control unit F3. Also, in the information processing device 1E, in the model storage unit F4, a plurality of candidates of the backbone model BB (not shown) are stored together with a plurality of candidates of the head model HD.

[0121] Note that the hardware configuration of the information processing apparatus 1E may be the same as that of the information processing apparatus 1, and redundant explanations are avoided. The processing of the control unit F3E described below can be realized by software processing by the CPU 11 (denoted as CPU 11E for distinction) included in the information processing apparatus 1E.

[0122] In the model storage unit F4, each candidate of the backbone model BB is a model that has been learned to be specialized for the assumed sensing environment. For example, a backbone model BB that has been learned to be specialized for a lighting environment with strong oblique irradiation light by the morning sun or the evening sun, a backbone model BB that has been learned to be specialized for a lighting environment with a substantially uniform illuminance distribution with little oblique irradiation light, and the like.

[0123] The control unit F3E has functions as an environment estimation unit F31, a model selection unit F32, and a deployment processing unit F33. The environment estimation unit F31 performs a process of estimating the sensing environment for the target sensing device 3 based on the attached information accumulated as the accumulation information 5a together with the feature amount data in the database 5.

[0124] Here, as exemplified above, the attached information includes a time stamp for the feature amount data, information on the sensing location, detection information of the temperature and motion sensors, illuminance information of the external light, information on the environmental sound detected by the microphone, weather information at the time of sensing, and the like. Such attached information can be paraphrased as environmental estimation basis information that can be used as an estimation basis for the sensing environment in the target sensing device 3.

[0125] For example, based on the above-mentioned attached information, the environment estimation unit F31 estimates whether the sensing environment of the target sensing device 3 is the environment with strong oblique light (oblique light environment) or the environment with a substantially uniform illuminance distribution (uniform illuminance environment) as described above. For this purpose, the environment estimation unit F31 in this example uses the time stamp and weather information in the attached information to estimate whether the current sensing environment is the above-mentioned oblique light environment or uniform illuminance environment. For example, if the weather is sunny during the morning or evening hours, it can be estimated that it is an oblique light environment. On the other hand, even during the morning or evening hours, if the weather is cloudy or rainy, it can be estimated as a uniform illuminance environment rather than an oblique light environment. By using multiple types of environment estimation basis information, the accuracy of environment estimation can be improved.

[0126] Note that the estimation process of the sensing environment mentioned here is just an example and is not limited to this. For example, the environment estimation can be not from two types of estimations but from three or more types of estimations. Also, regarding the information used for environment estimation, the above example is just an example, and information according to the estimation content can be appropriately selected and used as the environment estimation basis information.

[0127] The model selection unit F32 selects a backbone model BB corresponding to the sensing environment estimated by the environment estimation unit F31 from among a plurality of candidates of the backbone model BB stored in the model storage unit F14. In the figure, the backbone model BB selected by the model selection unit F32 is denoted as "backbone model BBx".

[0128] The deployment processing unit F33 controls so that the backbone model BB selected by the model selection unit F32 is deployed to the target sensing device 3. That is, the backbone model BBx is deployed to the sensing device 3 as the transmission source of the feature amount data to which the attached information used for environment estimation is added.

[0129] With the information processing apparatus 1E as the fourth embodiment as described above, it is possible to cause the target sensing apparatus 3 to use the backbone model BB suitable for the sensing environment. In other words, it is possible to cause the backbone model BB capable of performing appropriate feature amount extraction for the sensing environment to be used. Therefore, it is possible to improve the inference performance.

[0130] Here, in the above, as an example of realizing inference processing suitable for the environment, an example was given in which models obtained by learning specialized for each environment were prepared as candidates for the backbone model BB. However, by using the mechanism of deploying the backbone model BB selected according to the estimation environment as in the fourth embodiment to the sensing apparatus 3, it is also possible to meet a wide range of user needs. For example, on the premise that the head model HD to be used for each environment (e.g., human detection for the first head model HD in the morning time zone and vehicle detection for the second head model HD in the night time zone) is defined (by the user), a method of selecting a backbone model BB corresponding to the head model HD to be used in the estimated environment from the candidates may be considered.

[0131] In the fourth embodiment, the candidates for the backbone model BB are not limited to preparing models with different parameters, and it is also conceivable to prepare models with different depths (number of intermediate layers).

[0132] Further, in the fourth embodiment, the environment estimation basis information is not limited to those exemplified above. For example, it may include parameters of the image signal processing performed by the image signal processing unit 42 (e.g., brightness adjustment value, gamma correction value, white balance adjustment value, saturation and hue adjustment values, etc.) and imaging parameters of the imaging unit 41 (e.g., ISO sensitivity, shutter speed (exposure time), aperture value, etc.). It is also conceivable to include information such as the power consumption amount of the sensing apparatus 3.

[0133] Also, in the fourth embodiment, candidates for the backbone model BB may be stored in a storage device different from the candidates for the head model HD. Also, candidates for the backbone model BB may be stored outside the information processing apparatus 1E.

[0134] <5. Modification Example> Note that the embodiments are not limited to the specific examples described above, and configurations as various modification examples can be adopted. For example, in the above, an imaging device was exemplified as an example of the sensing device 3, but sensing in the present technology is not limited to image sensing (subject light sensing), and may also include other sensing such as sound sensing, acceleration sensing, and angular velocity sensing. For example, in the case of sound sensing, as inference processing for sensing data, examples include text conversion processing, speaker separation (separation of which utterance is by which speaker) processing, and summary processing of the utterance content. In the present technology, the sensing data may be any data that can be the target of inference processing by an AI model, and the sensing may be any method that can obtain such sensing data.

[0135] <6. Summary of Embodiments> As described above, the information processing apparatus (the same 1, 1A, 1B, 1C, 1D, 1E) as an embodiment has a sensor and uses a backbone model, which is an AI model that extracts feature amounts for input data, to extract feature amounts from sensing data obtained by a sensing apparatus that extracts feature amounts from sensing data by the sensor. From a database in which the feature amount data is stored, an input unit (the same F1) that inputs the feature amount data, a selection unit (the same F2) that selects a head model based on user input information from among a plurality of candidates for the head model, which is an AI model that performs inference processing using the feature amount data as input data, and a control unit (the same F3, F3A, F3B, F3C, F3D, F3E) that controls so that inference processing is performed using the feature amount data input by the input unit as input data using the head model selected by the selection unit. The feature data extracted by the backbone model from the sensing data has a reduced data volume compared to the sensing data. Further, the feature data is obtained by extracting the features of the sensing target from the sensing data and does not represent the sensing target itself. Therefore, according to the above configuration, when performing inference processing using an AI model based on the data accumulated in the database from the sensing device, it is possible to reduce the storage capacity of the database and the communication data volume between the sensing device and the database, while achieving privacy protection.

[0136] Also, in the information processing apparatus according to the embodiment, information indicating a target task to be realized using inference processing is input as user input information, and the selection unit selects the head model capable of executing the inference task specified from the target task. Thereby, it is possible to perform inference processing for realizing the target task desired by the user.

[0137] Furthermore, in the information processing apparatus according to the embodiment, the feature data is associated with attachment information indicating at least either the sensing time or the sensing location of the sensing data that is the extraction source of the feature data. The input information includes designation information for designating time or location, and the selection unit selects, as the feature data input by the input unit, the feature data corresponding to the time or location designated by the designation information from among the feature data stored in the database. Thereby, when the database stores feature data for sensing data at various times and locations, it is possible to perform inference processing targeting the sensing data at the time and location desired by the user.

[0138] Furthermore, in the information processing apparatus according to the embodiment, the input information is information representing the target task in a language format, and the selection unit selects the head model and the feature data based on the input information using a large language model. By using a large language model, it becomes possible to accurately search for head models and feature data that match the task conditions specified by the user in a language format. Therefore, the probability that the target task desired by the user is appropriately executed can be increased.

[0139] Also, in the information processing apparatus as an embodiment, the selection unit selects a plurality of head models with different inference tasks based on the input information, and the control unit inputs the same feature data input by the input unit to each of the head models selected by the selection unit as input data. The inference process is executed (see FIG. 9). Thereby, for the same sensing target, inference processes for different tasks can be performed simultaneously in parallel. Also, when performing inference processes for different tasks, only a single model may be used as the backbone model, and it is not necessary to perform the process of extracting features using the backbone model for each task on the sensing device side. Therefore, the processing load on the sensing device can be reduced.

[0140] Furthermore, the information processing apparatus as an embodiment (the same 1A, 1B, 1C) includes a presentation processing unit (the same F6) that performs a process of presenting, to the user, as visualization information, analysis information of a sensing target generated based on inference result information by a head model. Thereby, information indicating the result of the target task specified by the user can be presented to the user as visualization information.

[0141] Furthermore, in the information processing apparatus as an embodiment (the same 1A, 1B, 1C), the control unit (F3A, F3B, F3C) controls so that the head model selected by the selection unit is deployed to the sensing device. Thereby, when the target task specified by the user requires an inference process targeting data during sensing rather than data stored in the database, it becomes possible to cause the sensing device to execute an inference process targeting data during sensing. Therefore, the range of applicable tasks can be widened.

[0142] Also, in the information processing apparatus (the same 1A) as an embodiment, the selection unit selects a plurality of head models, and the control unit (the same F3A) controls such that the plurality of head models selected by the selection unit are deployed to the sensing device, and gives an instruction to the sensing device to switch the head model to be used among the deployed head models. Thereby, it becomes possible to cope with a case where it is required to switch and execute inference processes using different head models on the time axis for the inference process to be executed by the sensing device for the data during sensing. For example, when the selection unit selects a plurality of head models in response to a case where a plurality of head models are required to realize the target task of a single user, as the inference process for the data during sensing, it becomes possible to execute the inference processes by those head models in a time-division manner on one sensing device, and there is no need to deploy and execute each head model on a plurality of sensing devices, so that the number of sensing devices can be reduced. Alternatively, when the selection unit selects a plurality of head models for realizing the target tasks of respective users based on the input information from a plurality of users who require execution of different target tasks, since it becomes possible to execute the inference processes for the target tasks of respective users in a time-division manner on one sensing device, there is no need to prepare a sensing device for each user when realizing the target tasks of the plurality of users respectively, and the number of sensing devices can be reduced.

[0143] Furthermore, in the information processing apparatus (the same 1B) as an embodiment, the control unit (the same F3B) gives an instruction to the sensing device in which the head model is deployed to transfer the deployed head model to another sensing device. As a result, when it is required to perform inference processing based on the head model selected by the selection unit not only in a single sensing device but also in another sensing device, it becomes unnecessary to perform the deployment process of the head model to the other sensing device. Therefore, the processing load on the information processing device can be reduced.

[0144] Furthermore, in the information processing device (the same 1B) as an embodiment, the deployed head model is a head model that performs object detection processing for a specific object, and the control unit (the same F3B) issues an instruction indicating that, as an instruction, transfer to another sensing device of the deployed head model is performed on the condition that an inference result that a specific object has been detected is obtained by the inference processing by the head model. As a result, in the sensing device in which the head model is deployed, for example, when a specific object such as a criminal or a stolen vehicle is detected, it is possible to arrange for the head model for detecting the specific object to be transferred to surrounding other sensing devices, and it is possible to enable tracking of the specific object.

[0145] Also, in the information processing device (the same 1C) as an embodiment, the selection unit selects a plurality of head models, and the control unit (the same F3C) controls so that the plurality of head models selected by the selection unit are respectively deployed to different sensing devices, and issues an instruction to transmit the feature amount data extracted by the backbone model of one of the deployment destinations to the sensing devices of other deployment destinations to one of the deployment destination sensing devices. As a result, when a plurality of types of inference processing should be performed on one piece of sensing data, those inference processes can be realized as distributed processing using a plurality of sensing devices.

[0146] Furthermore, in the information processing apparatus as an embodiment (same as 1D), the control unit (same as F3D) performs execution control of relearning for the backbone model used by the sensing device in response to the establishment of a predetermined trigger condition, and controls so that the backbone model obtained by the relearning is deployed to the sensing device. Thereby, in response to a case where improvement in the configuration of the AI model is required in terms of, for example, inference accuracy or functionality, it becomes possible to perform relearning of the backbone model and deployment of the backbone model after relearning to the sensing device. Therefore, it is possible to improve the inference performance and to adapt to the required inference functions.

[0147] Furthermore, in the information processing apparatus as an embodiment, the predetermined trigger condition is a condition based on an evaluation of the accuracy of the inference process performed using the head model. Thereby, it becomes possible to perform relearning of the backbone model and deployment of the backbone model after relearning using a decrease in inference accuracy as a trigger condition. Therefore, it is possible to improve the inference performance.

[0148] Also, in the information processing apparatus as an embodiment (same as 1E), environmental estimation basis information that can be used as an estimation basis for the sensing environment of the sensing data from which the feature amount data is extracted is associated with the feature amount data, and the control unit (same as F3E) estimates the sensing environment based on the environmental estimation basis information, selects a backbone model corresponding to the estimated sensing environment from among a plurality of candidates for the backbone model, and controls so that the selected backbone model is deployed to the sensing device that is the transmission source of the feature amount data. Thereby, it becomes possible to cause the sensing device to use a backbone model suitable for the sensing environment. Therefore, it is possible to improve the inference performance.

[0149] In addition, as an information processing method according to an embodiment, an information processing apparatus has a sensor and uses a backbone model, which is an AI model for performing feature extraction on input data, to perform feature extraction on sensing data obtained by a sensing apparatus that performs feature extraction on sensing data by the sensor. From a database in which the obtained feature amount data is stored, the feature amount data is input, and among a plurality of candidates of a head model, which is an AI model that performs an inference process using the feature amount data as input data, a head model is selected based on user input information, and control is performed so that an inference process using the input feature amount data as input data is performed using the selected head model. Even with such an information processing method, the same operations and effects as those of the information processing apparatus according to the above-described embodiment can be obtained.

[0150] Here, as an embodiment, it is possible to consider a program that realizes functions as an input unit, a selection unit, a control unit, etc., described with reference to FIG. 8 and the like above, in, for example, a CPU, a DSP, etc., or a device including these. That is, the program according to the embodiment is a program readable by a computer device, which has a sensor and uses a backbone model, which is an AI model for performing feature extraction on input data, to perform feature extraction on sensing data obtained by a sensing apparatus that performs feature extraction on sensing data by the sensor. From a database in which the obtained feature amount data is stored, the feature amount data is input, and among a plurality of candidates of a head model, which is an AI model that performs an inference process using the feature amount data as input data, a selection of the head model based on user input information is made, and a program that causes a computer device to realize a function of controlling so that an inference process using the input feature amount data as input data is performed using the selected head model. With such a program, functions as the above-described input unit, selection unit, control unit, etc. can be realized in a device as an information processing apparatus.

[0151] The program as described above can be pre-recorded in an HDD (Hard Disc Drive), SSD (Solid State Drive) as a recording medium built into devices such as computer devices, or in a ROM in a microcomputer having a CPU, etc. Alternatively, it can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), Blu-ray Disc (registered trademark), magnetic disk, semiconductor memory, memory card, etc. Such a removable recording medium can be provided as so-called packaged software. Also, such a program can be installed from a removable recording medium into a personal computer or the like, or can be downloaded from a download site via a network such as a LAN or the Internet.

[0152] Also, according to such a program, it is suitable for providing a wide range of inference processing methods as embodiments. Various forms of information processing devices can be made to function as devices that implement the inference processing method of the present disclosure.

[0153] Also, the information processing device as an embodiment has a sensor, and from a database in which feature amount data obtained by a sensing device that performs feature amount extraction on sensing data by a sensor using a first AI model that performs feature amount extraction on input data is stored, an input unit that inputs the feature amount data, a selection unit that selects a second AI model based on user input information from among a plurality of candidates of the second AI model that performs inference processing with the feature amount data as input data, and a control unit that controls so that inference processing is performed using the second AI model selected by the selection unit with the feature amount data input by the input unit as input data. It can also be said that it is equipped with these components. In this case, the first AI model may be configured to have a third AI model that performs feature extraction and feature integration. This makes it possible to protect privacy while reducing the database storage capacity and the amount of communication data between the sensing device and the database, even when the backbone model BB has the above-mentioned bottleneck model NC.

[0154] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0155] <7. This Technology> The present technology can also be configured as follows. (1) an input unit that inputs feature amount data from a database that stores feature amount data obtained by a sensing device that has a sensor and extracts feature amounts from sensing data by the sensor using a backbone model that is an AI model that extracts feature amounts from input data; a selection unit that selects a head model from among a plurality of candidates of a head model, which is an AI model that performs inference processing using feature amount data as input data, based on information input by a user; a control unit that controls an inference process to be performed using the head model selected by the selection unit and the feature amount data input by the input unit as input data. Information processing device. (2) As the user's input information, information indicating a target task to be realized using the inference process is input; The selection unit is Select the head model capable of executing an inference task specified from the target task. The information processing device according to (1). (3) The feature amount data is associated with auxiliary information indicating at least one of a sensing time or a sensing location of the sensing data from which the feature amount data is extracted, The input information includes designation information for designating time or location, The selection unit, selects, as the feature amount data input by the input unit, the feature amount data corresponding to the time or location designated by the designation information from among the feature amount data stored in the database. The information processing apparatus according to (2) above. (4) The input information is information representing the target task in a language format, The selection unit, selects the head model and the feature amount data based on the input information using a large language model. The information processing apparatus according to (3) above. (5) The selection unit, selects a plurality of the head models with different inference tasks based on the input information, The control unit, causes each head model selected by the selection unit to execute an inference process using the same feature amount data input by the input unit as input data. The information processing apparatus according to any one of (2) to (4) above. (6) A presentation processing unit that performs processing to present, as visualization information, analysis information of a sensing target generated based on inference result information by the head model to a user. The information processing apparatus according to any one of (2) to (5) above. (7) The control unit, controls so that the head model selected by the selection unit is deployed to the sensing device. The information processing apparatus according to any one of (2) to (6) above. (8) The selection unit, selects a plurality of the head models, The control unit, controls so that the plurality of head models selected by the selection unit are deployed to the sensing device, and Instruct the sensing device to switch the head model to be used among the deployed head models The information processing apparatus according to (7) above (9) The control unit Instruct the sensing device in which the head model is deployed to transfer the deployed head model to another sensing device The information processing apparatus according to (7) or (8) above (10) The deployed head model is a head model that performs object detection processing for a specific object, The control unit As the instruction, give an instruction indicating that the transfer of the deployed head model to the other sensing device is performed on the condition that an inference result that the specific object has been detected by the inference processing using the head model is obtained The information processing apparatus according to (9) above (11) The selection unit selects a plurality of the head models, The control unit Control so that the plurality of head models selected by the selection unit are respectively deployed to different sensing devices, and Give an instruction to one of the deployment destination sensing devices to transmit the feature amount data extracted by the backbone model of the one sensing device to the other deployment destination sensing devices The information processing apparatus according to any one of (7) to (10) above (12) The control unit In response to the establishment of a predetermined trigger condition, perform execution control of re-learning for the backbone model used by the sensing device, and control so that the backbone model obtained by the re-learning is deployed to the sensing device The information processing apparatus according to any one of (1) to (11) above (13) The predetermined trigger condition is a condition based on an evaluation of the accuracy of the inference process performed using the head model The information processing apparatus according to (12) above (14) The feature amount data is associated with environment estimation basis information that can be used as an estimation basis for the sensing environment of the sensing data from which the feature amount data is extracted The control unit Estimates the sensing environment based on the environment estimation basis information, selects a backbone model corresponding to the estimated sensing environment from among a plurality of candidates of the backbone model, and controls so that the selected backbone model is deployed to the sensing device that is the transmission source of the feature amount data The information processing apparatus according to claim 1 (15) An information processing apparatus Has a sensor, and inputs the feature amount data from a database storing the feature amount data obtained by a sensing device that performs feature amount extraction on sensing data by a sensor using a backbone model, which is an AI model that performs feature amount extraction on input data Selects the head model based on the user's input information from among a plurality of candidates of the head model, which is an AI model that performs an inference process using the feature amount data as input data Controls so that an inference process using the input feature amount data as input data is performed using the selected head model An information processing method (16) A program readable by a computer device Has a sensor, and inputs the feature amount data from a database storing the feature amount data obtained by a sensing device that performs feature amount extraction on sensing data by a sensor using a backbone model, which is an AI model that performs feature amount extraction on input data Among a plurality of candidates of a head model, which is an AI model that performs an inference process using feature quantity data as input data, select the head model based on user input information, Cause the computer device to realize a function of controlling so that an inference process using the input feature quantity data as input data is performed using the selected head model Program. (17) A recording medium on which a program readable by a computer device is recorded, Input the feature quantity data from a database storing the feature quantity data obtained by a sensing device that has a sensor and performs feature quantity extraction on sensing data by the sensor using a backbone model, which is an AI model that performs feature quantity extraction on input data Among a plurality of candidates of a head model, which is an AI model that performs an inference process using feature quantity data as input data, select the head model based on user input information, A recording medium on which a program is recorded that causes the computer device to realize a function of controlling so that an inference process using the input feature quantity data as input data is performed using the selected head model Recording medium. (18) An input unit that inputs the feature quantity data from a database storing the feature quantity data obtained by a sensing device that has a sensor and performs feature quantity extraction on sensing data by the sensor using a first AI model that performs feature quantity extraction on input data, A selection unit that selects the second AI model based on user input information from among a plurality of candidates of the second AI model that performs an inference process using the feature quantity data as input data, A control unit that controls so that an inference process using the feature quantity data input by the input unit as input data is performed using the second AI model selected by the selection unit, and An information processing apparatus. (19) The first AI model has a third AI model that performs feature extraction and feature integration. The information processing apparatus according to (18) above.

Explanation of Signs

[0156] 1, 1A, 1B, 1C, 1D, 1E Information processing apparatus 2 Network 3 Sensing device 4 User terminal 5 Database 5a Stored information 11, 11A, 11B, 11C CPU 12 ROM 13 RAM 14 AI processing unit 30 Image sensor 31 Imaging optical system 32 Optical system drive unit 33 Camera control unit 34 Memory unit 35 Communication unit 36 Sensor unit 37 Bus 41 Imaging unit 42 Image signal processing unit 43 In-sensor control unit 44 AI processing unit 45 Memory unit BB, BB1, BB2, BB’, BBx Backbone model HD, HDh, HDc, HD1, HD2 Head model F1 Input unit F2 Selection unit F3, F3A, F3B, F3C, F3D, F3E Control unit F4 Model storage unit F5 Analysis unit F6 Presentation processing unit F7 Relearning processing unit F31 Environment estimation unit F32 Model selection unit F33 Deployment processing unit

Claims

1. an input unit that inputs feature amount data from a database that stores feature amount data obtained by a sensing device that has a sensor and extracts feature amounts from sensing data by the sensor using a backbone model that is an AI model that extracts feature amounts from input data; A selection unit that selects a head model from among a plurality of candidates of a head model, which is an AI model that performs inference processing using feature amount data as input data, based on information input by a user; a control unit that controls an inference process to be performed using the head model selected by the selection unit and the feature amount data input by the input unit as input data. Information processing device.

2. As the user's input information, information indicating a target task to be realized using the inference process is input; The selection unit is Select the head model capable of executing an inference task specified from the target task. The information processing device according to claim 1 .

3. The feature amount data is associated with auxiliary information indicating at least one of a sensing time or a sensing location of the sensing data from which the feature amount data is extracted, The input information includes designation information for designating a time or a place, The selection unit is The feature amount data corresponding to the time or place designated by the designation information is selected from the feature amount data stored in the database as the feature amount data to be input by the input unit. The information processing device according to claim 2 .

4. the input information is information expressing the target task in a linguistic form; The selection unit is The head model and the feature data are selected based on the input information using a large-scale language model. The information processing device according to claim 3 .

5. The selection unit is Selecting a plurality of head models having different inference tasks based on the input information; The control unit is The head model selected by the selection unit is caused to execute an inference process using the same feature amount data input by the input unit as input data. The information processing device according to claim 2 .

6. A presentation processing unit that performs processing to present analytical information of a sensing target generated based on inference result information by the head model to a user as visualized information. The information processing device according to claim 2 .

7. The control unit is Control so that the head model selected by the selection unit is deployed to the sensing device The information processing apparatus according to claim 2

8. The selection unit Selects a plurality of the head models The control unit Controls so that the plurality of head models selected by the selection unit are deployed to the sensing device, and Instructs the sensing device to switch the head model to be used among the deployed head models The information processing apparatus according to claim 7

9. The control unit Instructs the sensing device to which the head model is deployed to transfer the deployed head model to another sensing device The information processing apparatus according to claim 7

10. The deployed head model is a head model that performs object detection processing for a specific object, and The control unit As the instruction, gives an instruction indicating that the transfer of the deployed head model to the other sensing device is to be performed on the condition that an inference result that the specific object has been detected by the inference processing by the head model has been obtained The information processing apparatus according to claim 9

11. The selection unit selects a plurality of the head models, and The control unit Controls so that the plurality of head models selected by the selection unit are respectively deployed to different sensing devices, and Gives an instruction to one of the sensing devices as the deployment destination to transmit the feature amount data extracted by the backbone model of the one sensing device to the sensing devices at the other deployment destinations The information processing apparatus according to claim 7

12. The control unit Performs execution control of re-learning for the backbone model used by the sensing device in response to the establishment of a predetermined trigger condition, and controls so that the backbone model obtained by the re-learning is deployed to the sensing device The information processing apparatus according to claim 1

13. The predetermined trigger condition is a condition based on an evaluation of the accuracy of the inference processing performed using the head model The information processing apparatus according to claim 12

14. The feature amount data is associated with environment estimation basis information that can be used as an estimation basis for the sensing environment of the sensing data from which the feature amount data is extracted, and The control unit Estimate the sensing environment based on the environmental estimation basis information, select a backbone model corresponding to the estimated sensing environment from among a plurality of candidates of the backbone model, and control so that the selected backbone model is deployed to the sensing device that is the source of the feature amount data. The information processing apparatus according to claim 1.

15. The information processing apparatus Inputs the feature amount data from a database storing the feature amount data obtained by a sensing device that has a sensor and performs feature amount extraction on sensing data by the sensor using a backbone model which is an AI model that performs feature amount extraction on input data. Selects the head model based on the user input information from among a plurality of candidates of the head model which is an AI model that performs inference processing with the feature amount data as input data. Controls so that inference processing is performed using the selected head model with the input feature amount data as input data. An information processing method.

16. A program readable by a computer device, Inputs the feature amount data from a database storing the feature amount data obtained by a sensing device that has a sensor and performs feature amount extraction on sensing data by the sensor using a backbone model which is an AI model that performs feature amount extraction on input data. Selects the head model based on the user input information from among a plurality of candidates of the head model which is an AI model that performs inference processing with the feature amount data as input data. Causes the computer device to realize a function of controlling so that inference processing is performed using the selected head model with the input feature amount data as input data. Program.

17. A recording medium on which a program readable by a computer device is recorded, Inputs the feature amount data from a database storing the feature amount data obtained by a sensing device that has a sensor and performs feature amount extraction on sensing data by the sensor using a backbone model which is an AI model that performs feature amount extraction on input data. Selects the head model based on the user input information from among a plurality of candidates of the head model which is an AI model that performs inference processing with the feature amount data as input data. A program that causes the computer device to realize a function of controlling so that inference processing using the input feature amount data as input data is performed using the selected head model is recorded on the recording medium. **Claim 18** An input unit that inputs the feature amount data from a database in which the feature amount data obtained by a sensing device that has a sensor and performs feature amount extraction on sensing data by the sensor using a first AI model that performs feature amount extraction on input data is stored, a selection unit that selects the second AI model based on user input information from among a plurality of candidates of the second AI model that performs inference processing using the feature amount data as input data, and a control unit that controls so that inference processing using the feature amount data input by the input unit as input data is performed using the second AI model selected by the selection unit. An information processing apparatus. **Claim 19** The first AI model has a third AI model that performs feature amount extraction and feature amount integration. The information processing apparatus according to claim 18.

Images