Signal processing device, signal processing method, and program

Abstract

To facilitate real-time processing.SOLUTION: A signal processing device according to the present technology includes an inference processing unit that performs inference processing using an AI model, a control unit that receives designation information that designates at least a portion of the inference result obtained as a result of the inference processing and issues an output instruction for the inference result on the basis of the designation information, and an output unit that outputs the inference result on the basis of the output instruction.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This technology relates to the technical field of signal processing apparatuses, signal processing methods, and programs for performing inference results using an AI model.

Background Art

[0002] Techniques for obtaining recognition results by inputting predetermined input data into an AI model and performing predetermined recognition processing have been spreading. The AI model can be deployed on edge-side devices by miniaturization and weight reduction (see, for example, Patent Document 1 below). By deploying the AI model on an edge-side device, it becomes possible to perform recognition processing on the edge-side device, and only the recognition result is output to a subsequent device such as a server device, thereby suppressing the amount of data related to data communication.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in an edge-side device where an AI model is deployed, for example, in an image sensor, an interface is constructed so that predetermined recognition results and image data are output for each imaging frame from the start of the imaging operation. Therefore, a subsequent device that has received the recognition result by the AI model has difficulty in performing real-time processing because it analyzes the received recognition result, selects necessary information, and then uses it for various processes.

[0005] This technology has been made in view of such problems, and an object thereof is to easily perform real-time processing.

Means for Solving the Problems

[0006] The signal processing device according to the present technology includes an inference processing unit that performs inference processing using an AI model, a control unit that receives designation information for designating at least a part of the inference results obtained as a result of the inference processing, and issues an output instruction for the inference results based on the designation information, and an output unit that outputs the inference results based on the output instruction. As a result, not all of the inference results obtained by the inference processing are necessarily output to the subsequent stage, but a part of them can be output.

Brief Description of Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0008] Hereinafter, with reference to the accompanying drawings, embodiments of the information processing apparatus according to the present technology will be described in the following order. <1. Configuration of Signal Processing System> <2. Data Transfer> <3. Flow of Processing> <4. Modified Example> <5. Application Example> <6. Summary> <7. The Present Technology>

[0009] <1. Configuration of Signal Processing System> FIG. 1 is a block diagram showing a schematic configuration example of a signal processing system 1 as an embodiment according to the present technology. The signal processing system 1 is, for example, a camera device 1A having each part in one camera housing.

[0010] The camera device 1A is configured to include a plurality of AI processing devices that perform inference processing using an AI (Artificial Intelligence) model. Note that the inference processing using the AI model is simply referred to as "AI processing". That is, the AI processing device refers to a device that performs AI processing. Also, the AI processing on an image is referred to as "AI image processing".

[0011] The plurality of AI processing devices included in the camera device 1A are, for example, a signal processing device 2 and a front-stage device 3 that is a device in front of the signal processing device 2.

[0012] The front-stage device 3 is, for example, an image sensor 3A that receives light from a subject and performs photoelectric conversion to obtain a predetermined image.

[0013] The signal processing device 2 is configured as, for example, a single substrate on which an ISP (Image Signal Processor) capable of executing AI image processing is implemented.

[0014] The camera device 1A includes a subsequent-stage device 4 that is a device subsequent to the signal processing device 2. The subsequent-stage device 4 is, for example, a device that identifies inference results required outside the camera device 1A among various inference results obtained by AI image processing in the preceding-stage device 3 or the signal processing device 2 located in front of it, and transmits them to the preceding-stage device 3 or the signal processing device 2.

[0015] The subsequent-stage device 4 is, for example, a microcomputer such as an AP (Application Processor). The subsequent-stage device 4 may be mounted on a substrate different from the signal processing device 2 or may be mounted on the same substrate as the signal processing device 2.

[0016] The image sensor 3A as the preceding-stage device includes, for example, an imaging element unit 31, a developing processing unit 32, a control unit 33, a first AI processing unit 34, and a data transfer unit 35.

[0017] The imaging element unit 31 includes a pixel array unit 31a as a light-receiving unit and a readout circuit (not shown). The pixel array unit 31a is formed by two-dimensionally arranging pixels that output signals corresponding to the amount of received light by performing photoelectric conversion.

[0018] In the readout circuit, for the electrical signal obtained by photoelectric conversion, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. are executed, and further A / D (Analog / Digital) conversion processing is performed.

[0019] The image data output from the image sensor 3A and based on the digital signal read by the readout circuit is RAW image data.

[0020] The development processing unit 32 appropriately performs preprocessing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, etc. on the RAW image data after A / D conversion processing. In the preprocessing, clamp processing for clamping the black levels of R, G, and B to a predetermined level on the captured image signal, correction processing between the color channels of R, G, and B, etc. are performed. In the synchronization processing, color separation processing is performed so that the image data for each pixel has all the color components of R, G, and B. For example, in the case of an image sensor using a Bayer array color filter, demosaicing processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion processing, resolution conversion processing is executed on the image data subjected to various signal processes.

[0021] In the codec processing, for the image data subjected to the above various processes, for example, encoding processing for recording or communication, file generation is performed. In the codec processing, as the file format of the moving image, for example, file generation in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264 can be performed. Also, it is conceivable to perform file generation in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) as still image files.

[0022] The image data generated by the development processing unit 32 is used as the input image data input to the AI model.

[0023] Such a development processing unit 32 is provided, for example, in the image sensor 3A as an ISP (Image Signal Processor).

[0024] The control unit 33 causes the first AI processing unit 34 to execute AI image processing using the first AI model M1. Here, the AI image processing by the first AI processing unit 34 is described as "first AI image processing".

[0025] The control unit 33 also receives, as instruction information, information for selecting at least a part of the first inference results obtained by the first AI image processing from the subsequent device, specifically, from the subsequent device 4. The control unit 33 causes the first AI processing unit 34 to supply a predetermined first inference result to the data transfer unit 35 based on the received instruction information. Note that the control unit 33 may cause all of the first inference results obtained by the first AI processing unit 34 to be supplied to the data transfer unit 35, and instruct the data transfer unit 35 to transfer a part of the first inference results based on the instruction information to the subsequent stage.

[0026] The first AI processing unit 34 inputs the first input image data G1 to the first AI model M1 based on the instruction of the control unit 33, and obtains a first inference result. The first input image data G1 is, for example, image data generated by the developing processing unit 32.

[0027] The first input image data G1 input to the first AI model M1 may be image data subjected to various developing processes by the developing processing unit 32, RAW image data, or crop image data in which a predetermined region is further cut out by the developing processing unit 32.

[0028] The first AI model M1 executes, for example, a process of estimating a region in which a person appears in the first input image data G1 (see FIG. 2). Specifically, the first AI model M1 outputs classifying information, likelihood information, and coordinate information of the detected objects obtained by performing object recognition on the input image data.

[0029] Note that FIG. 2 is a diagram showing an image in which a recognized person in the first input image data G1 is surrounded by a dashed bounding box. FIG. 3 is a diagram showing an overview of the first inference result obtained by the first AI image processing using the first AI model M1.

[0030] Based on an instruction from the control unit 33, the first AI processing unit 34 supplies, for example, only the coordinate information among the classification information, the likelihood information, and the coordinate information to the data transfer unit 35.

[0031] Such a first AI processing unit 34 is provided, for example, in the image sensor 3A as a DSP (Digital Signal Processor) or the like.

[0032] The data transfer unit 35 transfers the first inference result selected based on an instruction from the control unit 33, for example, the coordinate information, to the subsequent signal processing device 2.

[0033] Data transfer between the image sensor 3A and the signal processing device 2 is performed, for example, in accordance with MIPI (Mobile Industry Processor Interface).

[0034] The transfer data compliant with MIPI includes a packet header, a data body, and a packet footer. The packet header is provided with a data type area for defining the data type for each packet.

[0035] The transfer data of MIPI includes a packet storing RAW image data by designating "RAW10" or the like in the data type area, a packet storing the first inference result by designating "embedded data" in the data type area, and the like.

[0036] The signal processing device 2 includes a receiving unit 21, a developing processing unit 22, a control unit 23, a second AI processing unit 24, and a data transfer unit 25.

[0037] The receiving unit 21 receives the first inference result compliant with MIPI and the image data from the front-stage image sensor 3A. The image data received by the receiving unit 21 from the image sensor 3A may be, for example, RAW image data or the first input image data G1 input to the first AI model M1.

[0038] The developing processing unit 22 is provided as an ISP or the like and performs developing processing on the RAW image data received from the receiving unit 21. When the image data on which the developing processing has been applied is received from the receiving unit 21, the execution of the developing processing is not essential. In this case, the developing processing unit 22 may function as a processing unit that performs image processing on image data other than the RAW image data.

[0039] Note that the image data generated by the developing processing unit 22 is used for AI image processing by the second AI processing unit 24.

[0040] The control unit 23 changes, for example, the parameters used for the developing processing performed by the developing processing unit 22 based on the first inference result received from the receiving unit 21. For example, when the inference accuracy of the first AI image processing executed in the front-stage image sensor 3A is low, the parameters for the developing processing are changed in order to improve the inference accuracy of the second AI image processing executed by the second AI processing unit 24 of the signal processing device 2.

[0041] Specifically, when the RAW image is an image taken under backlight and AI image processing for detecting a subject of a predetermined color is executed, the parameters for color correction are changed so as to return the color tone changed by the backlight to the original color tone. Also, when it is presumed that the inference accuracy of the first AI image processing has decreased due to the overall brightness of the image being too high, the parameters for the developing processing that affect the brightness of the image are changed.

[0042] The control unit 23 causes the second AI processing unit 24 to execute AI image processing using the second AI model M2. Here, the AI image processing by the second AI processing unit 24 is described as "second AI image processing".

[0043] The control unit 23 also receives, as instruction information, information for selecting at least a part of the second inference results obtained by the second AI image processing, from the subsequent device, specifically, the subsequent device 4. Based on the received instruction information, the control unit 23 causes the second AI processing unit 24 to supply a predetermined second inference result to the data transfer unit 25.

[0044] The second AI processing unit 24 is provided as a DSP or the like, and inputs the second input image data G2 to the second AI model M2 based on the instruction of the control unit 23 to obtain a second inference result.

[0045] The second input image data G2 input to the second AI model M2 is image data subjected to various development processes by the development processing unit 22, but may be crop image data obtained by cutting out a predetermined region based on the coordinate information obtained as the first inference result. Note that the second input image data G2 may be RAW image data.

[0046] The second AI model M2 identifies a face region and executes a face authentication process based on, for example, the second input image data G2 as a crop image in which a region where a person appears is cut out (see FIG. 4). Specifically, the second AI model M2 outputs ID (Identification) information, likelihood information, and coordinate information about the face region for the person identified as the face authentication result for the input image data.

[0047] Note that FIG. 4 is a diagram showing an image in which a bounding box indicating a face region used for face authentication with a broken line surrounds a crop image cut out from the first input image data G1, which is the second input image data G2 input to the second AI model M2. FIG. 5 is a diagram showing an overview of the second inference results obtained when the second AI image processing is performed for each person detected in the first AI image processing.

[0048] Note that the second input image data G2 does not necessarily have to be a cropped image. For example, the first input image data G1 and the second input image data G2 may be the same image.

[0049] Based on the instruction from the control unit 23, the second AI processing unit 24 supplies, for example, only the ID information among the ID information, the likelihood information, and the coordinate information to the data transfer unit 25.

[0050] The data transfer unit 25 transfers the second inference result selected based on the instruction from the control unit 23, for example, the ID information about the detected person, to the subsequent device 4 at the later stage.

[0051] The data transfer between the signal processing device 2 and the subsequent device 4 is performed, for example, in accordance with MIPI.

[0052] The ID information transferred in the data transfer from the signal processing device 2 to the subsequent device 4 is stored and transferred in a packet in which "embedded data" is specified in the data type area.

[0053] The subsequent device 4 includes at least a control unit 41. The control unit 41 specifies the inference result to be output from the camera device 1A and notifies the image sensor 3A and the signal processing device 2. As a result, specific first and second inference results are output from the image sensor 3A and the signal processing device 2 from the camera device 1A.

[0054] Note that these inference results may be output from the subsequent device 4. In that case, the subsequent device 4 includes a receiving unit 42 and a transmitting unit 43, and the receiving unit 42 receives the inference result selected from the image sensor 3A and the signal processing device 2, and the transmitting unit 43 may output the inference result to the outside of the camera device 1A.

[0055] When outputting image data from the camera device 1A, the receiving unit 42 may receive the image data from the image sensor 3A and the signal processing device 2, and the transmitting unit 43 may output the image data to the outside of the camera device 1A. Note that predetermined development processing may be performed on the image data received from the image sensor 3A or the signal processing device 2. In that case, a development processing unit 44 may be provided in the subsequent-stage device 4.

[0056] The receiving unit 42 receives MIPI-compliant packets from the signal processing device 2.

[0057] When the packet contains image data and development processing is required, the development processing unit 44 receives RAW image data or the like from the receiving unit 42 and performs the necessary development processing. Note that, in the development processing in the development processing unit 44, parameters adjusted by the control unit 41 may be used in order to suitably perform the processing content in the subsequent-stage device 4.

[0058] Note that the control unit 41 may perform processing using the inference result received from the image sensor 3A or the signal processing device 2.

[0059] For example, consider the case where it is used as a device that authenticates a person imaged by the camera device 1A and determines whether to unlock the door based on the authentication result. The control unit 41 of the subsequent-stage device 4 performs a process of collating the detected person with the person permitted to enter based on the second inference result acquired from the signal processing device 2 by referring to a database or the like, and determines whether to output an instruction to unlock the door lock based on the collation result.

[0060] As a result, the camera device 1A does not need to output not only the image data but also the inference result to the outside, and privacy can be protected.

[0061] Furthermore, in the subsequent-stage device 4, AI image processing may be performed using the image data developed by the development processing unit 44. For example, an AI processing unit may be provided in the subsequent-stage device 4, and AI image processing or the like may be performed in the AI processing unit.

[0062] <2. Data Transfer> As described above, data transfer between the image sensor 3A, the signal processing device 2, and the subsequent-stage device 4 is performed using, for example, MIPI-compliant packets.

[0063] The MIPI-compliant packet includes a packet header PH, a data section DS, and a packet footer PF (see FIG. 6).

[0064] The packet header PH further includes a data ID, a word count (WC), and an ECC (Error Correction Code). The data ID further consists of a 2-bit virtual channel VC and a 6-bit data type. In FIG. 6, only the data type of the packet header PH is shown. Note that "EBD" in the figure indicates embedded data as the data type.

[0065] Various data as shown in FIG. 6 can be output from the camera device 1A to an external device. Here, the image sensor 3A transfers only the information instructed by the control unit 41 of the subsequent-stage device 4 to the subsequent-stage signal processing device 2. Similarly, the signal processing device 2 outputs only the information instructed by the control unit 41 to the subsequent-stage device 4 or from the camera device 1A to an external device.

[0066] Then, the image sensor 3A and the signal processing device 2 can change the data to be transferred to the subsequent stage according to the situation.

[0067] For example, when installing the camera device 1A, as shown in FIG. 7, the control unit 41 instructs the image sensor 3A and the signal processing device 2 to transfer only the information regarding the image quality of the captured image. In the example shown in FIG. 7, only the imaging setting information and the result information obtained by analyzing the image after the development process are output from the camera device 1A by the control unit 41.

[0068] With these devices arranged outside the camera device 1A, it becomes easy to perform an optimal imaging setting according to the installation location of the camera device 1A while using this information.

[0069] Next, when deploying the AI model after installing the camera device 1A, as shown in FIG. 8, the control unit 41 instructs the image sensor 3A and the signal processing device 2 to transfer only the information about the first AI model M1 and the information about the second AI model M2. In the example shown in FIG. 8, only the first input image data G1, the first inference result, the second input image data G2, and the second inference result are output from the camera device 1A by the control unit 41.

[0070] For devices arranged outside the camera device 1A, it is possible to generate and adjust the AI model to be deployed to the camera device 1A while using this information.

[0071] Then, during operation using the camera device 1A, as shown in FIG. 9, the control unit 41 instructs the signal processing device 2 to transfer only the second inference result.

[0072] During operation, in a device arranged outside the camera device 1A, it is possible to determine whether to unlock the door or not using the second inference result. And by not outputting the image data outside the camera device 1A, it is possible to suitably protect privacy.

[0073] The information stored in the MIPI-compliant packet differs depending on the information specified by the control unit 41 as described above. Therefore, the size of the data stored in the data section DS of the packet also differs.

[0074] In packet transfer, it is possible to allow differences in the data size of the data section DS, or to unify the data size.

[0075] For example, as shown in FIG. 10, even if the packet sizes are different due to different data sizes of the data section DS, the packets may be transferred to the subsequent stage as they are.

[0076] Alternatively, as shown in FIG. 11, when the data sizes of the data portions DS are different, padding data PD may be used to fill the data portions DS so that the data sizes of the data portions DS become a predetermined size. In this case, the packet sizes to be transferred are unified.

[0077] <3. Flow of processing> An example of the processing executed by the image sensor 3A of the camera device 1A is shown in FIG. 12. The imaging element unit 31 of the image sensor 3A performs imaging processing by performing exposure control and readout processing in step S101. As a result, RAW image data as digital data is output from the imaging element unit 31.

[0078] The development processing unit 32 performs the above-described development processing in step S102. As a result, for example, RGB image data or the like is obtained.

[0079] The first AI processing unit 34 performs first AI image processing on the developed image data as the first input image data G1 in step S103.

[0080] The first AI processing unit 34 determines in step S104 whether an inference result has been obtained as a result of the first AI image processing, for example, whether a target subject has been detected.

[0081] When the first AI processing unit 34 determines that the target subject has been detected (step S104: Yes), it proceeds to step S105 and acquires the first inference result.

[0082] On the other hand, when the first AI processing unit 34 determines that the target subject has not been detected (step S104: No), it skips the processing of step S105 and proceeds to step S106.

[0083] The control unit 33 determines in step S106 whether information to be transferred from the image sensor 3A is specified. When it is determined that transfer information is specified (step S106: Yes), the control unit 33 instructs the data transfer unit 35 in step S107 to select only the specified information and store it in an MIPI-compliant packet.

[0084] On the other hand, when it is determined that transfer information is not specified (step S106: No), the control unit 33 instructs the data transfer unit 35 in step S108 to select all the information that can be output and store it in an MIPI-compliant packet. Note that when it is determined in step S104 that the first inference result cannot be obtained, information other than the first inference result, such as imaging setting information, is stored in the packet.

[0085] After executing either step S107 or step S108, the data transfer unit 35 performs a process of transferring the generated packet to the subsequent signal processing device 2 in step S109.

[0086] After finishing the process of step S109, the image sensor 3A executes the process of step S101 again. That is, in the image sensor 3A, a series of processes shown in FIG. 13 are executed for each imaging frame.

[0087] Subsequently, an example of the process executed by the signal processing device 2 of the camera device 1A is shown in FIG. 13. The receiving unit 21 of the signal processing device 2 acquires the first inference result from the previous image sensor 3A in step S201. Note that the receiving unit 21 may acquire information other than the first inference result, such as imaging setting information, in step S201, and the received information may not include the first inference result.

[0088] The control unit 23 performs a process of changing the parameters used for the development process in step S202. Note that this process is not essential and may be executed, for example, when the second input image data G2 for performing high-precision inference cannot be appropriately generated.

[0089] The developing processing unit 22 performs developing processing in step S203. Note that, instead of performing developing processing on the RAW image data in step S203, the developing processing unit 22 may perform image processing on image data other than the RAW image data. Also, when inputting the RAW image data as the second input image data G2 to the second AI model M2, the processing in step S203 is not essential.

[0090] The second AI processing unit 24 performs second AI image processing using the image data after image processing as the second input image data G2 in step S204.

[0091] In step S205, the second AI processing unit 24 determines whether an inference result has been obtained as a result of the second AI image processing, for example, whether the target subject has been detected.

[0092] When the second AI processing unit 24 determines that the target subject has been detected (step S205: Yes), it proceeds to step S206 to obtain the second inference result.

[0093] On the other hand, when the second AI processing unit 24 determines that the target subject has not been detected (step S205: No), it skips the processing in step S206 and proceeds to step S207.

[0094] In step S207, the control unit 23 determines whether information transferred from the signal processing device 2 is specified. When it is determined that the transfer information is specified (step S207: Yes), the control unit 23 instructs the data transfer unit 25 in step S208 to select only the specified information and store it in an MIPI-compliant packet.

[0095] On the other hand, when it is determined that the transfer information is not specified (step S207: No), the control unit 23 instructs the data transfer unit 25 in step S209 to select all the information that can be output and store it in an MIPI-compliant packet.

[0096] After executing either step S208 or step S209, in step S210, the data transfer unit 25 performs a process of transferring (transmitting) the generated packet to the subsequent-stage subsequent device 4 or a device external to the camera device 1A.

[0097] After finishing the process of step S210, in the signal processing device 2, the process of step S201 is executed again.

[0098] <4. Variation> In the above description, the camera device 1A equipped with one image sensor 3A was given as an example of the signal processing system 1. As shown in FIG. 14, the signal processing system 1 in this variation is a camera device 1B equipped with a plurality of image sensors 3B, 3C, and 3D.

[0099] The image sensors 3B, 3C, and 3D have the same configuration as the image sensor 3A. In FIG. 14, the configurations of the image sensors 3C and 3D are omitted.

[0100] Instruction information about the transfer data is supplied from the control unit 41 of the subsequent device 4 to the control units 33 of the image sensors 3B, 3C, and 3D.

[0101] From the data transfer units 35 of the image sensors 3B, 3C, and 3D, the transfer data corresponding to the instruction information is stored in packets in the MIPI standard and transferred to the signal processing device 2.

[0102] The signal processing device 2 receives the transfer data from each of the image sensors 3B, 3C, and 3D at the receiving unit 21 and performs a second inference process in the second AI processing unit 24.

[0103] The signal processing device 2 also selects transfer data based on instructions from the control unit 23 from the image sensors 3B, 3C, 3D and various types of information generated by the signal processing device 2, and the selected data is transmitted from the data transfer unit 25 to the subsequent device 4 or a device external to the camera device 1B.

[0104] Note that the image sensors 3B, 3C, 3D may be different types of image sensors. For example, the image sensor 3B may be an RGB sensor that generates a color image, the image sensor 3C may be a ToF (Time of Flight) sensor that generates a distance image, and the image sensor 3D may be a thermal sensor that generates a temperature image.

[0105] Information generated by these image sensors 3B, 3C, 3D, such as various types of image data and inference results, can be appropriately selected by the signal processing device 2 to perform suitable processing in subsequent processing.

[0106] In the above example, an example of selecting the type of transfer data for each phase of the camera device 1A such as during installation or operation was described, but different data may be selected and transferred for each frame.

[0107] For example, until a person is detected, the inference result in the image sensor 3D as a thermal sensor is mainly output from the camera device 1B, and when a subject with a surface temperature equal to or higher than a predetermined temperature is detected, the inference result in the image sensor 3B as a color image sensor is output from the camera device 1B. Various usage methods are conceivable.

[0108] In the above, an example of transferring packet data compliant with MIPI as transfer data was shown, but data compliant with SPI (Serial Peripheral Interface) may also be transferred, or data compliant with a parallel interface may be transferred.

[0109] <5. Application Example> In the above example, an example was described in which face authentication is performed on a person imaged by the camera device 1A (1B), and whether or not to unlock the door lock is determined according to the result. However, the application of the present technology is not limited to this.

[0110] For example, even in the same face authentication system, the posture of a person is detected by AI image processing by the image sensor 3A (3B), and only the information of a person who is taking a posture of bringing the face close to the camera device 1A for face authentication is transferred to the subsequent signal processing device 2. The signal processing device 2 performs face detection processing as AI image processing, and transfers the result information of face detection to the subsequent subsequent device 4. At this time, the signal processing device 2 may specify the way light hits the subject according to the recognition result of the image sensor 3A, and optimize the parameters of the development process. In the subsequent device 4 or an external device of the camera device 1A, further AI image processing is performed, face authentication processing is performed, and who the subject is is specified. At this time, the subsequent device 4 may optimize the parameters of the development process in the same manner as the signal processing device 2, and the optimization process may be performed by receiving the parameters from the signal processing device 2.

[0111] In addition to the face authentication system, a vehicle license plate recognition system can be considered. Specifically, the vehicle is detected by AI image processing by the image sensor 3A, the signal processing device 2 detects the license plate of the detected vehicle, and the subsequent device 4 or an external device of the camera device 1A performs a process of identifying the vehicle and its owner by collating the license plate information of the vehicle. In the collation process of the vehicle license plate information, it is further determined whether or not the collation result matches the specified vehicle, and when it matches, information indicating that the vehicle to be detected has been captured by the camera device 1A as a surveillance camera may be output. Regarding the development process in each device, the parameter optimization process may be performed in the same manner as in the previous face authentication system. Also, for example, the parameters of the development process may be changed according to the color of the vehicle body of the detected vehicle.

[0112] Alternatively, the signal processing system 1 may be a system that reads a two-dimensional code or the like. For example, the image sensor 3A performs a process of detecting an object. In the subsequent signal processing device 2, a two-dimensional code attached or printed on the detected object is detected. In the further subsequent stage 4, a process of reading the detected two-dimensional code is performed to verify whether an appropriate two-dimensional code is assigned to the object. Regarding the development process in each device, optimization processing of parameters may be appropriately performed in the same manner as in the previous face recognition system, license plate recognition system, etc.

[0113] In addition to this, in the image sensor 3A, a dog is detected. When a dog is detected, a person approaching the dog is detected in the subsequent signal processing device 2. When these are detected, a process of performing an audio announcement may be executed in the subsequent device 4 considering that the person is a visually impaired person.

[0114] Note that the examples given here are just examples, and various other configurations are conceivable.

[0115] <6. Summary> As described in the various examples above, the signal processing device 2 in the signal processing system 1 includes an inference processing unit (second AI processing unit 24) that performs inference processing (e.g., second AI image processing) using an AI model (second AI model M2), a control unit 23 that receives designation information specifying at least a part of the inference result (second inference result) obtained as a result of the inference processing, and issues an output instruction for the inference result based on the designation information, and an output unit (data transfer unit 25) that outputs the inference result based on the output instruction. As a result, not all of the inference results obtained by the inference processing are necessarily output to the subsequent stage, but a part of them can be output. Therefore, since only the inference results specified according to the subsequent processing content can be output, it is not necessary to analyze the recognition results in the subsequent stage (e.g., the subsequent device 4), and real-time processing can be easily performed. In addition, by outputting only the necessary inference results, the data transfer load can be reduced, and it becomes possible to support a low data transfer rate.

[0116] As described with reference to FIG. 1 and the like, in the signal processing device 2, a receiving unit 21 that receives, as a first inference result, the result of a first inference process (for example, first AI image processing) using the first AI model M1 from a preceding device 3 (for example, image sensor 3A) is provided, and the control unit 23 may issue an output instruction based on designation information that designates predetermined information from among a second inference result, which is an inference result of a second inference process (for example, second AI image processing) that is an inference process by an inference processing unit (second AI processing unit 24), and the first inference result. As a result, it becomes possible to select and output only the inference results necessary for the processing of a subsequent stage (for example, subsequent device 4) from among the plurality of inference results obtained by a plurality of inference processes. Therefore, even if the processing of the subsequent stage is changed, it becomes possible to output an appropriate inference result corresponding to the changed requirements, and it becomes possible to reduce the processing burden of the subsequent stage. In addition, for example, by executing inference processing in a distributed manner among a plurality of devices, such as performing person detection in the preceding device 3 and face detection in the subsequent signal processing device 2, it becomes possible to perform inference processing with high inference accuracy even on an edge device with low performance. And by using a plurality of devices, it becomes possible to perform appropriate inference processing in accordance with the requirements in various edge devices. Note that the “preceding device 3” as used herein includes not only an independent single device but also a microcomputer that performs preceding processing and a substrate on which a predetermined IC (Integrated Circuit) is mounted. That is, the preceding device 3 and the signal processing device 2 may be included in one housing. In addition, in that case, the preceding device 3 may be an image sensor 3A, and the signal processing device 2 may be a microcomputer. That is, the preceding device 3 and the signal processing device 2 may be provided in one device.

[0117] As described with reference to FIGS. 1 to 3 and the like, the receiving unit 21 of the signal processing device 2 receives image data (such as RGB image data or RAW image data) together with the first inference result from the previous-stage device 3 (for example, the image sensor 3A), and the inference processing unit (the second AI processing unit 24) may perform second inference processing (for example, second AI image processing) by the second AI model M2 based on the image data. The image data received by the signal processing device 2 may be, for example, the image data input to the first AI model M1, or may be RAW image data captured by the previous-stage device 3 or the like. By receiving the image data from the previous-stage device 3, it becomes possible to perform inference processing on the image in the signal processing device 2. For example, it becomes possible to perform person detection in the previous-stage first AI model M1 and face detection in the subsequent-stage second AI model M2. Therefore, even an edge-side device with poor processing capabilities can perform advanced inference processing.

[0118] As described with reference to FIG. 1 and the like, the signal processing device 2 includes an image processing unit (for example, the developing processing unit 22) that generates second input image data G2 to be input to the second AI model M2 by performing image processing on the image data, and the inference processing unit (the second AI processing unit 24) may perform second inference processing by inputting the second input image data G2 to the second AI model M2. Thereby, an input image to be input to the second AI model M2 again is generated in the signal processing device 2. Therefore, it becomes possible to generate appropriate second input image data G2 as the image data input to the second AI model M2, and it becomes possible to improve the accuracy of the inference processing (for example, second AI image processing) in the second AI model M2.

[0119] As described with reference to FIG. 1 and the like, the receiving unit 21 in the signal processing device 2 receives RAW image data as the image data, and the image processing unit (for example, the developing processing unit 22) may generate the second input image data G2 by performing developing processing as the image processing. When the signal processing device 2 receives RAW image data, it becomes easy for the signal processing device 2 to generate appropriate second input image data G2 to be input to the second AI model M2.

[0120] As described with reference to FIG. 1 and the like, the control unit 23 of the signal processing device 2 supplies parameters based on the first inference result to an image processing unit (for example, the development processing unit 22), and the image processing unit may perform image processing based on the parameters. Thereby, it becomes possible to change the image processing as development processing in the subsequent-stage signal processing device 2 according to the first inference result of the inference processing (for example, the first AI image processing) in the preceding-stage device 3 (for example, the image sensor 3A). For example, when receiving the likelihood information of the first inference result in the preceding-stage device 3 and the likelihood information is lower than a predetermined value, the parameters used for the development processing for suitably performing the second inference processing (for example, the second AI image processing) by the second AI model M2 can be changed. Also, when the subject to be detected in the inference processing in the preceding-stage device 3 cannot be detected, by changing the parameters used for the development processing for generating the second input image data G2 input to the subsequent-stage second AI model M2, it becomes possible to obtain an appropriate second inference result in the second AI model M2. Note that an output instruction may be given to the preceding-stage device 3 so that information related to parameter adjustment is transferred from the preceding-stage device 3 to the signal processing device 2. Thereby, for example, information such as the brightness information of the image and the way light hits the face of the subject is transferred from the preceding-stage device 3 to the signal processing device 2 as appropriate. Thereby, it becomes possible to optimize the AI image processing in the signal processing device 2.

[0121] As described with reference to FIG. 14 and the like, a plurality of preceding-stage devices 3 (for example, image sensors 3B, 3C, 3D) are provided in the signal processing device 2, the receiving unit 21 receives the first inference result for each of the plurality of preceding-stage devices 3, and the control unit 23 may give an output instruction based on the second inference result and designated information for designating predetermined information from among the plurality of first inference results. For example, in a plurality of front-end devices 3, an inference process for detecting a person (first AI image process) and an inference process for inferring a posture (first AI image process) are performed. Then, in the subsequent signal processing device 2, based on these detection results, for example, an area where a person is detected and an area where a face is imaged from the inference result of the posture are estimated, and a face authentication can be performed by inputting a crop image of the area into the second AI model M2. In this way, by using various inference results, the amount of computation of the inference process (for example, second AI image process) in the subsequent signal processing device 2 can be reduced, and the specifications required for the signal processing device 2 can be lowered. Also, by configuring the subsequent device 4 located further downstream of the signal processing device 2 to be able to transmit not only the second inference result but also the first inference result, a wide range of inference results required in the inference process of the subsequent device 4 can be selected and output. Therefore, it becomes possible to appropriately perform the inference process in the subsequent device 4.

[0122] As described with reference to FIG. 1 and the like, the front-end device 3 for the signal processing device 2 may be an image sensor 3A. Thereby, a first inference process (for example, first AI image process) and a second inference process (for example, second AI image process) can be performed as inference processes using the image captured by the image sensor. Therefore, it becomes possible to perform various inference processes without transmitting the image data to the downstream of the signal processing device 2, which is preferable from the viewpoint of privacy. In particular, when the signal processing device 2 is a microcomputer provided in the housing of the camera device 1A (1B), it is also possible to output only the inference result (first inference result or second inference result) without outputting the image outside the camera, and it becomes possible to suitably protect privacy.

[0123] As described with reference to FIG. 1 and the like, the output unit (data transfer unit 25) of the signal processing device 2 may output the inference result (first inference result or second inference result) in a format compliant with MIPI. By using MIPI designed on the premise that image data is stored, it is suitable for transmitting image data to a subsequent device. At that time, by transmitting only the specified inference result along with it, the burden of data transfer can be reduced.

[0124] As described with reference to FIG. 10 and the like, in data transfer in the signal processing device 2, the packet size in a format compliant with MIPI may be variable. Thereby, when transmitting the inference result (first inference result or second inference result) as embedded data, there is no need to worry about the data amount of the inference result.

[0125] As described with reference to FIG. 11 and the like, in data transfer in the signal processing device 2, the packet size in a format compliant with MIPI may be fixed. Thereby, when transmitting the inference result (first inference result or second inference result) as EBD, on the receiving side, packet data of a fixed size is received, so that the efficiency of the receiving process can be improved.

[0126] As described with reference to FIG. 1 and the like, the control unit 23 of the signal processing device 2 may receive specified information from a subsequent device 4 that receives output data from an output unit (data transfer unit 25). It becomes possible to output only necessary information from the inference result (first inference result or second inference result) in response to an instruction from the subsequent device 4. Therefore, it becomes possible to realize suitable data transfer for the subsequent device 4. Note that the “subsequent device 4” here includes not only an independent single device but also a microcomputer or the like that performs subsequent processing. That is, the subsequent device 4 and the signal processing device 2 may be included in one housing. For example, the above-described preceding device 3 may be an image sensor 3A, and the signal processing device 2 and the subsequent device 4 may be different microcomputers. That is, the preceding device 3, the signal processing device 2, and the subsequent device 4 may be provided in one camera device 1A (1B).

[0127] The signal processing method of the present technology is a method executed by a computer device, including: inference processing (e.g., second AI image processing) using an AI model (second AI model M2); a process of receiving designation information that designates at least a part of the inference result (second inference result) obtained as a result of the inference processing; a process of giving an output instruction regarding the inference result based on the designation information; and a process of outputting the inference result based on the output instruction.

[0128] The program of the present technology is a program for causing an arithmetic processing device to execute, including a function of performing inference processing (e.g., second AI image processing) using an AI model (second AI model M2); a function of receiving designation information that designates at least a part of the inference result (second inference result) obtained as a result of the inference processing; a function of giving an output instruction regarding the inference result based on the designation information; and a function of outputting the inference result based on the output instruction.

[0129] With such a signal processing method and program, the various operations and effects described above can also be obtained.

[0130] Such a program can be pre-recorded in an HDD (Hard Disk Drive) as a recording medium built into a device such as a computer device, or in a ROM (Read Only Memory) in a microcomputer having a CPU (Central Processing Unit). Alternatively, the program can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, CD-ROM (Compact Disk Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), Blu-ray Disc (registered trademark), magnetic disk, semiconductor memory, memory card, etc. Such a removable recording medium can be provided as so-called packaged software. In addition to installing such a program from a removable recording medium into a personal computer or the like, it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0131] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0132] Also, the above examples can be combined in any way, and various effects described above can be obtained even when using various combinations.

[0133] <7. The present technology> The present technology can also adopt the following configuration. (1) An inference processing unit that performs inference processing using an AI model, A control unit that receives designation information specifying at least a part of the inference results obtained as a result of the inference processing, and issues an output instruction for the inference results based on the designation information, An output unit that outputs the inference results based on the output instruction, and A signal processing device. (2) A receiving unit that receives, as a first inference result, the result of first inference processing using a first AI model from a preceding device, The control unit issues the output instruction based on the designation information that specifies predetermined information from among the second inference result that is the inference result of second inference processing that is the inference processing by the inference processing unit and the first inference result. The signal processing device according to (1) above. (3) The receiving unit receives image data from the preceding device together with the first inference result, The inference processing unit performs the second inference processing by a second AI model based on the image data. The signal processing device according to (2) above. (4) An image processing unit that generates second input image data to be input to the second AI model by performing image processing on the image data is provided. The inference processing unit performs the second inference processing by inputting the second input image data to the second AI model. The signal processing device according to (3) above. (5) The receiving unit receives RAW image data as the image data. The image processing unit generates the second input image data by performing development processing as the image processing. The signal processing device according to (4) above. (6) The control unit supplies parameters based on the first inference result to the image processing unit. The image processing unit performs the image processing based on the parameters. The signal processing device according to any one of (4) to (5) above. (7) A plurality of the preceding-stage devices are provided. The receiving unit receives the first inference result for each of the plurality of preceding-stage devices. The control unit issues the output instruction based on the specified information that instructs predetermined information from among the second inference result and the plurality of first inference results. The signal processing device according to any one of (2) to (6) above. (8) The preceding-stage device is an image sensor. The signal processing device according to any one of (2) to (7) above. (9) The output unit outputs the inference result in a format compliant with MIPI. The signal processing device according to any one of (1) to (8) above. (10) The packet size of the format compliant with MIPI is variable. The signal processing device according to (9) above. (11) The packet size of the format compliant with MIPI is fixed. The signal processing device according to the above (9). (12) The control unit receives the specified information from a subsequent device that receives output data from the output unit The signal processing device according to any one of the above (1) to the above (11). (13) (1) Inference processing using an AI model, (2) Processing for receiving specified information that designates at least a part of the inference results obtained as a result of the inference processing, (3) Processing for giving an output instruction regarding the inference results based on the specified information, (4) Processing for outputting the inference results based on the output instruction, which is executed by a computer device Signal processing method. (14) (1) A function for performing inference processing using an AI model, (2) A function for receiving specified information that designates at least a part of the inference results obtained as a result of the inference processing, (3) A function for giving an output instruction regarding the inference results based on the specified information, (4) A function for outputting the inference results based on the output instruction, which is executed by an arithmetic processing unit Program.

Explanation of Signs

[0134] 2 Signal processing device 21 Receiver 22 Development processing unit (image processing unit) 23 Control unit 24 Second AI processing unit (inference processing unit) 25 Data transfer unit (output unit) 3 Preceding device 3A Image sensor (preceding device) 3B Image sensor (preceding device) 3C Image sensor (preceding device) 3D Image sensor (preceding device) 4 Subsequent device G2 Second input image data M1 First AI model M2 Second AI Model (AI Model)

Claims

1. An inference processing unit that performs inference processing using an AI model, a control unit that receives designation information specifying at least a part of the inference results obtained as a result of the inference processing, and gives an output instruction regarding the inference results based on the designation information, and an output unit that outputs the inference results based on the output instruction, provided with a signal processing device.

2. A signal processing device according to claim 1, further comprising a reception unit that receives, as a first inference result, the result of first inference processing using a first AI model from a preceding-stage device, wherein the control unit gives the output instruction based on the designation information that designates predetermined information from among a second inference result that is the result of second inference processing which is the inference processing by the inference processing unit and the first inference result.

3. The signal processing device according to claim 2, wherein the reception unit receives image data together with the first inference result from the preceding-stage device, and the inference processing unit performs the second inference processing using a second AI model based on the image data.

4. The signal processing device according to claim 3, further comprising an image processing unit that generates second input image data to be input to the second AI model by performing image processing on the image data, wherein the inference processing unit performs the second inference processing by inputting the second input image data to the second AI model.

5. The signal processing device according to claim 4, wherein the reception unit receives RAW image data as the image data, and the image processing unit generates the second input image data by performing development processing as the image processing.

6. The signal processing device according to claim 4, wherein the control unit supplies parameters based on the first inference result to the image processing unit, and the image processing unit performs the image processing based on the parameters.

7. The signal processing device according to claim 2, wherein a plurality of the preceding-stage devices are provided, the reception unit receives the first inference result for each of the plurality of preceding-stage devices, and the control unit gives the output instruction based on the designation information that designates predetermined information from among the second inference result and the plurality of first inference results.

8. The signal processing device according to claim 2, wherein the preceding-stage device is an image sensor.

9. The signal processing device according to claim 1, wherein the output unit outputs the inference results in a format compliant with MIPI.

10. The signal processing device according to claim 9, wherein the packet size of the format compliant with MIPI is variable.

11. ​ ​ The packet size in the format compliant with the MIPI is fixed The signal processing device according to claim 9

12. The control unit receives the specified information from a subsequent device that receives output data from the output unit The signal processing device according to claim 1

13. Inference processing using an AI model, Processing for receiving specified information that specifies at least a part of the inference results obtained as a result of the inference processing, Processing for giving an output instruction regarding the inference results based on the specified information, Processing for outputting the inference results based on the output instruction, which is executed by a computer device Signal processing method

14. A function for performing inference processing using an AI model, A function for receiving specified information that specifies at least a part of the inference results obtained as a result of the inference processing, A function for giving an output instruction regarding the inference results based on the specified information, A function for outputting the inference results based on the output instruction, which is executed by an arithmetic processing unit Program

Images