Information processing device, imaging device, control method, and control program

Abstract

Provided are an information processing device, an imaging device, a control method, and a control program that make it possible to preserve image data that is highly evidential, while improving convenience.　The present invention comprises a processor (4), and if an evidence saving mode in which captured images are recorded as evidence has been set, the processor (4) outputs first image data obtained without the execution of first AI processing and second image data obtained with the execution of the first AI processing. The output destination of the first image data and the output destination of the second image data are different.

Description

Information processing device, imaging device, control method, and control program

[0001] The present invention relates to an information processing device, an imaging device, a control method, and a control program.

[0002] Patent Document 1 describes a surveillance system that is comprised of a network camera, a recording device, a viewing device, and a network, in which the network camera sends captured "low-resolution video" to the network, the recording device receives and records the low-resolution video, and also performs super-resolution processing on the low-resolution video to generate and record "high-resolution video," and further records an "encrypted signature" that is generated to link the two videos, and the viewing device displays the low-resolution video and the high-resolution video, and by encrypting and linking the two images, even if low-resolution video is converted into high-resolution video using super-resolution processing, the latter is not necessarily the original, thereby eliminating the problem of low evidentiary value.

[0003] Patent Document 2 describes an imaging device that includes a control unit, a detection unit, a tracking control unit, a "DL tracking unit" that tracks a subject using deep learning (DL), and a "non-DL tracking unit" that tracks a subject without using DL, and the control unit acquires one of the operation modes set according to the type and movement of the subject detected by the detection unit and the background, and notifies the tracking control unit, and for each operation mode, sets situations in which the DL tracking unit and the non-DL tracking unit are enabled, and by selectively using the DL tracking unit and the non-DL tracking unit according to the tracked target, power consumption is reduced in scenes where there is little need to obtain good tracking results.

[0004] Patent Document 3 describes an information processing system that includes a server device and multiple cameras, each of which has the function of performing AI image processing; when the server device acquires the necessary monitoring information from the cameras, it performs a process to determine whether the camera's usage state is an acceptable usage state, and if it determines that the camera's usage state is not an acceptable usage state, it disables the AI ​​image processing in the camera.

[0005] Japanese Patent Publication No. 2010-219889 Japanese Patent Publication No. 2023-086274 International Publication No. 2023 / 090037

[0006] One embodiment of the technique of the present disclosure provides an information processing device, an imaging device, a control method, and a control program that are capable of saving image data with high evidential value while improving convenience.

[0007] (1) An information processing device having a processor, wherein, when a first mode is set, the processor outputs first image data obtained by not executing a first AI process and second image data obtained by executing the first AI process, and the output destination of the first image data and the output destination of the second image data are different.

[0008] (2) The information processing device according to (1), wherein the first image data is output to a recording unit, and the second image data is output to a display unit.

[0009] (3) The information processing device according to (1) or (2), wherein when the first mode is set, the processor outputs a mixture of the first image data and the second image data.

[0010] (4) The information processing device according to (3), wherein the processor alternately outputs the mixed first image data and the mixed second image data.

[0011] (5) The information processing device according to any one of (1) to (4), wherein the processor sets the first mode when a first subject is detected.

[0012] (6) The information processing device according to any one of (1) to (5), wherein the first AI processing includes at least processing using a trained model.

[0013] (7) The information processing device according to any one of (1) to (6), wherein the first image data and the second image data are stream data.

[0014] (8) An information processing device according to any one of (1) to (7), wherein, when a second mode different from the first mode is set, the processor outputs the second image data out of the first image data and the second image data.

[0015] (9) An information processing device according to any one of (1) to (8), wherein, when the first mode is set, the processor executes a first processing that is a non-AI processing on imaging data, outputs the first image data on which the first AI processing has not been executed for the image data on which the first processing has been executed, and outputs the second image data on which the first AI processing has been executed for the image data on which the first processing has been executed.

[0016] (10) The information processing device according to any one of (1) to (9), wherein, when the first mode is set, the processor executes a second process, which is a non-AI process, on the first image data and the second image data.

[0017] (11) The information processing device according to any one of (1) to (10), wherein the processor generates, for at least the first image data, supplementary information related to the first AI processing.

[0018] (12) The information processing device according to (11), wherein the first image data is stream data, and the additional information indicates whether or not the first AI processing has been performed on the first image data over time.

[0019] (13) The information processing device according to any one of (1) to (12), wherein the processor performs processing to include, in at least the second image data, information visually indicating whether or not AI processing has been performed.

[0020] (14) An imaging device including the information processing device according to any one of (1) to (13).

[0021] (15) A control method for an information processing device having a processor, wherein, when a first mode is set, the processor outputs first image data obtained by not executing a first AI process and second image data obtained by executing the first AI process, and the output destination of the first image data and the output destination of the second image data are different.

[0022] (16) A control program for an information processing device having a processor, the control program causing the processor to execute a process to output, when a first mode is set, first image data obtained by not executing a first AI process and second image data obtained by executing the first AI process, and the output destination of the first image data and the output destination of the second image data are different.

[0023] (17) An information processing device having a processor, wherein, when a first mode is set, the processor outputs third image data having a first image area that is a non-execution area of ​​a first AI processing and a second image area that is an execution area of ​​the first AI processing.

[0024] (18) The information processing device according to (17), in which the processor sets the first mode when a first subject is detected.

[0025] (19) The information processing device according to (18), in which the first image area includes the first subject.

[0026] (20) The information processing device according to any one of (17) to (19), wherein the processor performs processing to include information regarding a boundary between the first image area and the second image area in the image data.

[0027] (21) An imaging device including the information processing device according to any one of (17) to (20).

[0028] (22) A control method for an information processing device having a processor, wherein, when a first mode is set, the processor outputs image data having a first image area that is a non-execution area of ​​a first AI processing and a second image area that is an execution area of ​​the first AI processing.

[0029] (23) A control program for an information processing device having a processor, the control program causing the processor to execute a process of outputting image data having a first image area that is a non-execution area of ​​a first AI processing and a second image area that is an execution area of ​​the first AI processing when a first mode is set.

[0030] According to the present invention, it is possible to provide an information processing device, an imaging device, a control method, and a control program that are capable of saving image data with high evidential value while improving convenience.

[0031] 1 is a diagram showing the configuration of the imaging device 1 of the first embodiment. FIG. 1 is a functional block diagram of a processing unit constituted by a processor 4 in a non-evidence preservation mode. FIG. 2 is a diagram showing the output of image data when the non-evidence preservation mode is set. FIG. 3 is a functional block diagram of a processing unit constituted by a processor 4 in an evidence preservation mode. FIG. 4 is a diagram showing the output of image data when the evidence preservation mode is set. FIG. 5 is a diagram showing a modified example of the output of image data in the evidence preservation mode. FIG. 6 is a flowchart showing the process of automatically setting the imaging mode. FIG. 7 is a diagram showing an example of incidental information indicating whether a first AI process is performed or not. FIG. 8 is a diagram showing a modified example of incidental information indicating whether a first AI process is performed or not. FIG. 9 is a diagram showing an example of a first image based on the first image data and a second image based on the second image data. FIG. 10 is a diagram showing the configuration of the imaging device 10 of the second embodiment. FIG. 11 is a functional block diagram of a processing unit constituted by a processor 14. FIG. 12 is a diagram showing the output of image data in a non-evidence preservation mode of the second embodiment. FIG. 13 is a diagram showing an example of a third image based on the third image data. FIG. 14 is a diagram showing an example of recognizing a first image area 91 in a third image 90. FIG. 15 is a flowchart showing the process of automatically setting the imaging mode in the second embodiment.

[0032] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0033] [Embodiment 1] <Imaging Device> Fig. 1 is a diagram showing the configuration of an imaging device 1 according to embodiment 1. As shown in Fig. 1, the imaging device 1 includes an imaging lens 2, an imaging element 3, an information processing device 4A, a recording unit 5, and a display unit 6. The information processing device 4A includes a processor 4. The imaging device 1 is applied to, for example, a surveillance camera.

[0034] The imaging lens 2 is a lens that forms an image of incident light on the imaging element 3. The imaging lens 2 is provided with, for example, an aperture that can mechanically adjust the amount of incident light. In this example, a single imaging lens 2 is shown, but multiple imaging lenses may be provided.

[0035] The imaging element 3 is configured by, for example, a CCD (Charge Coupled Device) element or a CMOS (Complementary Metal Oxide Semiconductor) element. The imaging element 3 captures an image of a target subject at a predetermined frame rate. The "predetermined frame rate" refers to, for example, several tens to several hundreds of frames per second. The imaging element 3 also converts incident light into an electrical signal by photoelectric conversion, converts the converted electrical signal into a digital signal, and outputs the converted digital signal to the processor 4 as imaging data. "Imaging data" is data obtained by the imaging element 3.

[0036] The processor 4 performs demosaic processing on the imaging data output from the imaging element 3 to convert it into full-color image data. The processor 4 also performs processes on the converted image data, such as sensor offset correction, white balance correction, AI (Artificial Intelligence) processing, color space conversion, and color correction. "AI processing" refers to processing using a trained model. The "trained model" refers to a machine learning model generated by prior learning processing (e.g., deep learning) based on training data. Processing using a trained model includes, for example, noise reduction, super-resolution, heat haze removal, HDR, haze removal, and blur removal. The processor 4 outputs the processed imaging data to the recording unit 5 as, for example, first image data and to a display unit 6, which is different from the recording unit 5, as second image data. The first image data and the second image data are stream data.

[0037] The recording unit 5 is, for example, a non-volatile memory provided inside or outside the imaging device 1. The recording unit 5 records the first image data output from the processor 4. The display unit 6 is, for example, a display unit formed of a liquid crystal display or the like provided inside or outside the imaging device 1. The display unit 6 displays an image based on the second image data output from the processor 4.

[0038] The imaging device 1 can switch the setting of whether or not to record captured surveillance images in the recording unit 5. Surveillance images are images that are recorded in the recording unit 5 as evidence images that have legal effect, for example. An imaging mode that is set to record as evidence images will be referred to as an "evidence preservation mode" below. An imaging mode that is not set to record as evidence images will be referred to as a "non-evidence preservation mode" below. The evidence preservation mode is an example of a "first mode" in the present invention. The non-evidence preservation mode is an example of a "second mode" in the present invention, and is the normal mode of the imaging modes of the imaging device 1.

[0039] The evidence preservation mode can be set by a user operation. Alternatively, the evidence preservation mode may be set automatically by an automatic setting function of the imaging device 1. The imaging device 1 sets the evidence preservation mode, for example, when a first subject is detected. The first subject is, for example, a predetermined specific subject (such as a person or a flying object). The imaging device 1 may detect the first subject by, for example, image recognition (which may use AI), or by radar or an infrared sensor.

[0040] <Functional Block in Non-Evidence Preservation Mode> Fig. 2 is a functional block diagram in non-evidence preservation mode of the processing unit configured by the processor 4. As shown in Fig. 2, the processing unit of the processor 4 includes a pre-processing unit 41 that does not perform AI processing, a first image processing unit 42 that does not perform AI processing, a second image processing unit 43 that performs AI processing, and a post-processing unit 44 that does not perform AI processing.

[0041] The preprocessing unit 41 performs non-AI preprocessing on the image data output from the image sensor 3. "Non-AI preprocessing" refers to preprocessing that does not involve AI processing. Examples of preprocessing include sensor offset correction and white balance correction. The preprocessing is an example of the "first processing" of the present invention. In the non-evidence storage mode, the preprocessing unit 41 outputs the preprocessed image data to the second image processing unit 43 but does not output it to the first image processing unit 42.

[0042] The first image processing unit 42 can perform first image processing without first AI processing on image data that has been preprocessed by the preprocessing unit 41. "First image processing without first AI processing" refers to first image processing without first AI processing. The first image processing unit 42 can output first image data obtained by performing the first image processing without first AI processing to the postprocessing unit 44. "First AI processing" refers to processing that uses at least a trained model (e.g., noise reduction, super-resolution, heat haze removal, HDR, haze removal, blur removal, etc.) and that includes processing that reduces the reliability or evidential admissibility of image data. "First image data without first AI processing" refers to image data obtained without performing first AI processing on a captured image. "First image data" is evidential image data that has not been subjected to first AI processing. Note that algorithm-based processing (e.g., super-resolution and noise reduction) may be performed instead of first AI processing.

[0043] However, in the non-evidence saving mode, in which the captured image is not set to be saved as an evidence image as in this example, image data is not input to the first image processing unit 42 from the pre-processing unit 41. Therefore, the first image processing unit 42 does not perform the first image processing. Therefore, in the non-evidence saving mode, the first image data is not output from the first image processing unit 42.

[0044] The second image processing unit 43 performs second image processing, which involves performing first AI processing, on the image data that has been pre-processed by the pre-processing unit 41. "Second image processing, which involves performing first AI processing," refers to second image processing that involves first AI processing. The second image processing unit 43 outputs second image data that has been obtained by performing the second image processing and that has been subjected to the first AI processing to the post-processing unit 44. "Second image data that has been subjected to first AI processing" is image data that has been obtained by performing the first AI processing on a captured image. The "second image data" is image data that has been subjected to processing that reduces the reliability and evidentiary admissibility of the image data, but has improved visibility compared to the first image data.

[0045] The post-processing unit 44 performs non-AI post-processing on the second image data output from the second image processing unit 43. Post-processing includes, for example, color space conversion and color correction. The post-processing is an example of the "second processing" in the present invention. The post-processing unit 44 outputs the second image data that has been post-processed to the display unit 6. Since the first image data is not output from the first image processing unit 42 to the post-processing unit 44, the post-processing unit 44 does not output image data to the recording unit 5.

[0046] When the non-evidence preservation mode is set, the processor 4 outputs to the display unit 6 the second image data obtained by executing the first AI processing, out of the first image data obtained by not executing the first AI processing and the second image data obtained by executing the first AI processing.

[0047] 3 is a diagram showing the output of image data when the non-evidence storage mode is set. When the imaging mode of the imaging device 1 is set to the "non-evidence storage mode," as described above, the second image processing unit 43 of the processor 4 outputs the second image data, and the first image processing unit 42 does not output the first image data.

[0048] 3 , the processor 4 performs pre-processing in the pre-processing unit 41 for each imaging frame 52 in imaging data 51 captured by the imaging device 1, and then performs second image processing in which the second image processing unit 43 executes first AI processing on each pre-processed image data 53. The processor 4 performs post-processing in the post-processing unit 44 for each second image data 54 that has been subjected to the second image processing, and then outputs each post-processed second image data 55 to the display unit 6. The processor 4 then displays a second image 56 based on the second image data 55 on the display unit 6. The second image 56 is an image with good visibility that has been subjected to the first AI processing.

[0049] In the non-evidence preservation mode, the processor 4 does not perform the first image processing, which does not involve the first AI processing by the first image processing unit 42, on each piece of image data 53 that has been preprocessed by the preprocessing unit 41. Therefore, the first image data is not output, and the first image data is not recorded in the recording unit 5.

[0050] <Functional Block in Evidence Preservation Mode> Fig. 4 is a functional block diagram in evidence preservation mode of the processing section constituted by the processor 4. As shown in Fig. 4, the processor 4 has the same configuration as Fig. 2, including a pre-processing section 41 that does not perform AI processing, a first image processing section 42 that does not perform AI processing, a second image processing section 43 that performs AI processing, and a post-processing section 44 that does not perform AI processing.

[0051] 4 is an imaging mode set to retain the captured image as an evidence image, so the processor 4 outputs the first image data that has not been subjected to the first AI processing to the recording unit 5, and outputs the second image data that has been subjected to the first AI processing to the display unit 6. Therefore, the evidence storage mode differs from the non-evidence storage mode in that the first image data that has not been subjected to the first AI processing is output to the recording unit 5.

[0052] As shown in FIG. 4, the preprocessing unit 41 outputs the image data that has undergone preprocessing to the second image processing unit 43 and also to the first image processing unit 42 .

[0053] The first image processing unit 42 performs first image processing without performing first AI processing on the image data that has been pre-processed by the pre-processing unit 41, and outputs the first image data obtained by performing the first image processing without performing the first AI processing to the post-processing unit 44.

[0054] As in the case of Figure 2, the second image processing unit 43 performs second image processing, which involves performing first AI processing, on the image data that has been pre-processed by the pre-processing unit 41, and outputs the second image data that has been subjected to the first AI processing obtained by performing the second image processing to the post-processing unit 44.

[0055] The post-processing unit 44 performs non-AI post-processing on the first image data output from the first image processing unit 42 and the second image data output from the second image processing unit 43. Then, the post-processing unit 44 outputs the post-processed first image data to the recording unit 5, and outputs the post-processed second image data to the display unit 6. The first image data and the second image data are output in a multi-stream.

[0056] 5 is a diagram showing the output of image data when the evidence preservation mode is set. When the imaging mode of the imaging device 1 is set to the "evidence preservation mode," as described above, the first image processing unit 42 of the processor 4 outputs the first image data, and the second image processing unit 43 outputs the second image data.

[0057] 5 , the processor 4 performs pre-processing in the pre-processing unit 41 on each imaging frame 52 in imaging data 51 captured by the imaging device 1, and then performs first image processing in the first image processing unit 42 on each piece of pre-processed image data 53. The processor 4 performs post-processing in the post-processing unit 44 on each piece of first image data 57 that has been subjected to the first image processing, and then outputs each piece of post-processed first image data 58 to the recording unit 5. The processor 4 then records a first image 59 based on the first image data 58 in the recording unit 5. The first image 59 has not been subjected to the first AI processing, and is an image that may have legal effect.

[0058] Furthermore, the processor 4 performs pre-processing in the pre-processing unit 41 for each imaging frame 52 in the imaging data 51 captured by the imaging device 1, and then performs second image processing in which the second image processing unit 43 executes first AI processing on each pre-processed image data 53. The processor 4 performs post-processing in the post-processing unit 44 for each second image data 54 that has been subjected to the second image processing, and then outputs each post-processed second image data 55 to the display unit 6. The processor 4 then causes the display unit 6 to display a second image 56 based on the second image data 55. The second image 56 is an image with good visibility that has been subjected to the first AI processing.

[0059] 6 is a diagram showing a modified example of the output of image data in the evidence preservation mode. When the imaging mode of the imaging device 1 is set to the "evidence preservation mode," the processor 4 may output a mixture of the first image data from the first image processing unit 42 and the second image data from the second image processing unit 43. For example, the processor 4 may alternately output the first image data from the first image processing unit 42 and the second image data from the second image processing unit 43. "Alternate output" means, for example, outputting the first image data and the second imaging data for each frame.

[0060] Specifically, as shown in Figure 6, the processor 4 performs preprocessing in the preprocessing unit 41 for each imaging frame 52 in the imaging data 51 captured by the imaging device 1, and then performs first image processing in the first image processing unit 42 without performing first AI processing on the odd-numbered image data 53 in the preprocessed image data 53, and performs second image processing in the second image processing unit 43 with first AI processing on the even-numbered image data 53.

[0061] The processor 4 performs post-processing in the post-processing unit 44 on each piece of first image data 57 that has been subjected to the first image processing, and then outputs each piece of post-processed first image data 58 to the recording unit 5. Then, the processor 4 causes the recording unit 5 to record a first image 59 based on the first image data 58.

[0062] Furthermore, the processor 4 performs post-processing on each piece of second image data 54 that has undergone the second image processing in the post-processing unit 44, and then outputs each piece of post-processed second image data 55 to the display unit 6. Then, the processor 4 causes the display unit 6 to display a second image 56 based on the second image data 55.

[0063] Note that the alternate output of the first image data and the second imaging data is not limited to alternate output for each frame. The alternate output may be, for example, alternate output of the first image data and the second imaging data for multiple frames. The alternate output may also be alternate output for different numbers of frames, such as outputting two frames of the first image data and one frame of the second image data.

[0064] <Image Capture Mode Setting Process> FIG. 7 is a flowchart showing the process of automatically setting the image capture mode.

[0065] It is assumed that the imaging mode of the imaging device 1 is set to the non-evidence preservation mode (evidence preservation mode=OFF) during image capture (step S11).

[0066] The processor 4 of the imaging device 1 performs, for example, image recognition based on the captured image, and executes a subject detection process to detect a specific subject (step S12). The specific subject may be, for example, a person or a flying object.

[0067] The processor 4 determines whether or not a specific subject has been detected in the subject detection process of step S12 (step S13).

[0068] If no specific subject is detected in step S13 (step S13: No), the processor 4 sets the imaging mode of the imaging device 1 to the non-evidence preservation mode (evidence preservation mode=OFF) (step S14), and then returns to step S12 to repeat each process. Note that if the imaging mode has already been set to the non-evidence preservation mode, the processor 4 maintains the non-evidence preservation mode.

[0069] If a specific subject is detected in step S13 (step S13: Yes), the processor 4 sets the imaging mode of the imaging device 1 to evidence preservation mode (evidence preservation mode=ON) (step S15), and then returns to step S12 to repeat each process. Note that if the imaging mode has already been set to the evidence preservation mode, the processor 4 maintains that evidence preservation mode.

[0070] In this example, the case where the evidence preservation mode is automatically set has been described, but the present invention is not limited to this. For example, when the imaging device 1 detects a specific subject, the imaging device 1 may notify the user of the detection, and the user may switch the evidence preservation mode ON / OFF.

[0071] <Additional information indicating whether AI processing is being performed / not being performed> Fig. 8 is a diagram showing an example of additional information indicating whether the first AI processing is being performed or not. As shown in Fig. 8, video data 71 and additional information 72 related to the video data 71 are recorded in the recording unit 5.

[0072] Image data with a time stamp is stored in the video data 71. The image data stored in the video data 71 is first image data obtained without executing the first AI processing. The first image data is stream data.

[0073] The accompanying information 72 stores time-stamped AI processing presence / absence information. The timestamp attached to the AI ​​processing presence / absence information is associated with the timestamp of the first image data stored in the video data 71. The AI ​​processing presence / absence information is accompanying information that indicates whether the first AI processing has been executed on the image data.

[0074] The additional information 72 is information indicating whether the first AI processing is performed or not performed on the first image data over time. "Whether the first AI processing is performed or not performed over time" may mean, for example, whether the first AI processing is performed or not performed for each time (the time indicated by the timestamp) or whether the first AI processing is performed or not for each frame of the first image data.

[0075] The processor 4 generates, for at least the first image data, incidental information 72 indicating whether the first AI processing is to be performed or not. The incidental information 72 in this example is recorded in the recording unit 5 as data separate from the first image data stored in the video data 71.

[0076] 9 is a diagram showing a modified example of incidental information indicating whether the first AI processing is performed or not. As shown in FIG. 9, video data 73 is recorded in the recording unit 5. Image data with a timestamp is stored in the video data 73 together with AI processing presence / absence information (incidental information) indicating whether the first AI processing is performed on the image data. The image data stored in the video data 73 is the first image data obtained without performing the first AI processing. In this way, the incidental information may be recorded in the recording unit 5 together with the first image data as data included in the video data 73.

[0077] <Image Based on Image Data> FIG. 10 is a diagram showing an example of a first image based on the first image data and a second image based on the second image data.

[0078] The first image 81 based on the first image data is an image recorded in the recording unit 5, and is an image that has not been subjected to the first AI processing. The first image 81 is an image that is recorded in an evidence preservation mode that is set when a specific subject 83 (e.g., a person) is detected.

[0079] The second image 82 based on the second image data is an image displayed on the display unit 6, and is an image that has been subjected to the first AI processing. The second image 82 is an image that is displayed both in the evidence preservation mode when a specific subject 83 is detected and in the non-evidence preservation mode when a specific subject 83 is not detected.

[0080] The processor 4 performs processing to include, in at least the first image data, information that visually indicates whether the first AI processing is being performed or not. The "processing to include information that visually indicates whether the first AI processing is being performed or not" refers to image processing that displays an image based on the first image data, allowing a person viewing the image to determine whether the image has been subjected to AI processing.

[0081] As shown in FIG. 10, a frame 84 of, for example, a specific color is additionally displayed on a first image 81 based on the first image data, indicating that the image has not been subjected to the first AI processing.

[0082] FIG. 11 is a diagram showing a modified example of a first image based on the first image data and a second image based on the second image data.

[0083] The first image 81 is an image based on the first image data, similar to Fig. 10, and is an image that has not been subjected to the first AI processing and is recorded in the recording unit 5 in the evidence preservation mode. The second image 82 is an image based on the second image data, similar to Fig. 10, and is an image that has been subjected to the first AI processing and is displayed on the display unit 6 in the evidence preservation mode and non-evidence preservation mode.

[0084] As shown in Figure 11, a message image 85 (e.g., "AI not used") indicating that the image has not undergone the first AI processing is additionally displayed on the first image 81 based on the first image data.

[0085] As described above, when the evidence preservation mode is set, the imaging device 1 of the first embodiment outputs the first image data obtained without performing the first AI processing and the second image data obtained with performing the first AI processing, outputting the first image data to the recording unit 5 and the second image data to the display unit 6. According to this configuration, when the imaging mode is set to the evidence preservation mode, the imaging device 1 can output not only the first image data that has not been subjected to the first AI processing, but also the second image data that has been subjected to the first AI processing. Therefore, the second image 56 with good visibility based on the second image data that has been subjected to the first AI processing can be displayed and viewed on the display unit 6, and the first image 59 as a legally effective evidence image based on the first image data that has not been subjected to the first AI processing can be recorded in the recording unit 5. This allows the second image 56 with good visibility to be used, while preserving the first image 59 that has high evidential value.

[0086] Furthermore, the imaging device 1 alternately outputs the first image data obtained without performing the first AI processing to the recording unit 5 and the second image data obtained by performing the first AI processing to the display unit 6. According to this configuration, the first image processing for obtaining the first image data without performing the first AI processing and the second image processing for obtaining the second image data by performing the first AI processing can be performed alternately, for example, for each frame of imaging data, and there is no need to perform the first image processing and the second image processing for each frame, thereby reducing data consumption.

[0087] Furthermore, when the imaging device 1 detects a predetermined specific subject (such as a person or a flying object), it sets the evidence preservation mode and outputs the first image data obtained without executing the first AI processing and the second image data obtained by executing the first AI processing, and when the specific subject is not detected, it outputs only the second image data obtained by executing the first AI processing. This configuration reduces the processing load of the processor 4 and also reduces data consumption.

[0088] Furthermore, the imaging device 1 generates additional information related to the first AI processing of the first image data over time, thereby enabling even more useful information to be added to the first image data recorded as evidence images having legal effect.

[0089] 12 is a diagram showing the configuration of an imaging device 10 according to embodiment 2. The imaging device 10 includes an imaging lens 12, an imaging element 13, an information processing device 14A, a recording unit 15, and a display unit 16. The information processing device 14A includes a processor 14.

[0090] As shown in Fig. 12 , the processor 14 of the second embodiment outputs the processed imaging data to the recording unit 15 and the display unit 16, for example, as third image data. In this respect, it differs from the processor 4 of the first embodiment, which outputs the processed imaging data to the recording unit 5 as first image data and to the display unit 6 as second image data, as shown in Fig. 1 . Note that the configurations and functions of the imaging lens 12, the imaging element 13, the recording unit 15, and the display unit 16 are similar to the configurations and functions of the respective components of the first embodiment described above. Also, although Fig. 12 shows the third image data being output to the display unit 16 as well, it is not necessary to output it to the display unit 16.

[0091] Similar to the imaging device 1 of the first embodiment, the imaging device 10 can switch the imaging mode between an "evidence preservation mode" in which the captured surveillance image is recorded in the recording unit 15, and a "non-evidence preservation mode" in which the captured surveillance image is not recorded. The imaging device 1 can be set to the evidence preservation mode by an automatic setting function, for example, when a first subject (such as a person or a flying object) is detected.

[0092] <Functional Block> Fig. 13 is a functional block diagram of a processing unit configured by the processor 14. As shown in Fig. 13, the processing unit of the processor 14 includes a pre-processing unit 141 that does not perform AI processing, an image processing unit 142 that performs AI processing, and a post-processing unit 143 that does not perform AI processing.

[0093] The pre-processing unit 141 performs non-AI pre-processing on the imaging data output from the imaging element 13. The pre-processing unit 141 outputs the image data that has undergone the pre-processing to the image processing unit 142.

[0094] The image processing unit 142 is capable of executing image processing that performs first AI processing on image data that has been preprocessed by the preprocessing unit 141. When the image processing unit 142 detects, for example, a first subject (a person, a flying object, etc.) in the image data that has been preprocessed, it sets the imaging mode to evidence preservation mode and executes image processing that does not perform the first AI processing and image processing that performs the first AI processing. "Image processing that does not perform AI processing" refers to image processing that does not perform AI processing. "Image processing that performs AI processing" refers to image processing that performs AI processing.

[0095] When the image processing unit 142 detects a first subject (such as a person or a flying object) in the image data, it sets a first image area to include the first subject. Furthermore, the image processing unit 142 sets an area other than the first image area as a second image area. The image processing unit 142 performs image processing without performing the first AI processing on the image data of the first image area that includes the first subject, and performs image processing with the first AI processing on the image data of the second image area that does not include the first subject.

[0096] The image processing unit 142 outputs third image data having image data of the first image area that has been subjected to image processing without performing the first AI processing and image data of the second image area that has been subjected to image processing with the first AI processing to the post-processing unit 143. The image data of the first image area that has not been subjected to the first AI processing is image data that can be used as an evidence image. The image data of the second image area that has been subjected to the first AI processing is image data that has improved visibility compared to the image data of the first image area.

[0097] The post-processing unit 143 performs non-AI post-processing on the third image data output from the image processing unit 142. The post-processing unit 143 outputs the post-processed third image data to the recording unit 15 and the display unit 16.

[0098] If the first subject (a person, a flying object, etc.) is not detected, the imaging device 10 sets the imaging mode to the non-evidence preservation mode using the automatic setting function. If the first subject (a person, a flying object, etc.) is not detected in the image data on which preprocessing has been performed, the image processing unit 142 performs only image processing that executes the first AI processing, and does not perform image processing that does not execute the first AI processing.

[0099] 14 is a diagram showing the output of image data in the non-evidence storage mode of embodiment 2. When the imaging mode of the imaging device 10 is set to the "non-evidence storage mode," as described above, the image processing unit 142 of the processor 14 performs only image processing, that is, executes the first AI processing on the imaging data.

[0100] 14 , the processor 14 performs image processing in which the preprocessing unit 141 preprocesses each imaging frame 152 in imaging data 151 captured by the imaging device 10, and then the image processing unit 142 executes first AI processing on each preprocessed image data 153. The image processing unit 142 executes the first AI processing on the entire captured image.

[0101] The processor 14 performs post-processing in the post-processing unit 143 on each piece of third image data 154 that has undergone image processing using the first AI processing, and then outputs each piece of post-processed third image data 155 to the recording unit 15 and the display unit 16. The processor 14 then causes the recording unit 15 to record a third image 156 based on the third image data 155. The processor 14 also causes the display unit 16 to display the third image 156 based on the third image data 155. When the non-evidence storage mode is set, the third image 156 is an image with good visibility in which the first AI processing has been performed on the entire image.

[0102] 15 is a diagram showing the output of image data in the evidence preservation mode of embodiment 2. When the imaging mode of the imaging device 10 is set to the "evidence preservation mode", as described above, the image processing unit 142 of the processor 14 performs image processing on the imaging data without performing the first AI processing, and image processing with performing the first AI processing.

[0103] 15 , the processor 14 performs preprocessing on each imaging frame 152 in imaging data 151 captured by the imaging device 10 using the preprocessing unit 141, and then performs image processing without the first AI processing and image processing with the first AI processing on each preprocessed image data 153 using the image processing unit 142. The processor 14 performs image processing without the first AI processing on image data of a first image region that includes a first subject (such as a person or a flying object), and performs image processing with the first AI processing on image data of a second image region that does not include the first subject. Specifically, the processor 14 performs image processing without the first AI processing on image data of a first image region 91 that is, for example, a region in which a person is detected and is surrounded by a dotted line rectangle, and performs image processing with the first AI processing on image data of a second image region 92 other than the first image region 91.

[0104] The processor 14 performs post-processing in the post-processing unit 143 on each of the third image data 154 that has been subjected to image processing without performing the first AI processing and image processing with performing the first AI processing, and then outputs the post-processed third image data 155 to the recording unit 15 and the display unit 16. The processor 14 then causes the recording unit 15 to record a third image 156 based on the third image data 155. The processor 14 also causes the display unit 16 to display the third image 156 based on the third image data 155. When the evidence storage mode is set, the third image 156 is an image that includes an image of the first image area 91 that has not been subjected to the first AI processing and an image of the second image area 92 that has been subjected to the first AI processing.

[0105] In this example, the third image including an image on which the first AI processing has not been performed and an image on which the first AI processing has been performed is displayed on the display unit 16. However, this is not limiting. For example, an image on which the first AI processing has been performed on the entire image may be separately generated, and the entire image on which the first AI processing has been performed and which has good visibility may be displayed on the display unit 16.

[0106] 16 is a diagram showing an example of a third image 90 based on the third image data. The third image 90 based on the third image data is an image recorded in the recording unit 15 and also displayed on the display unit 16.

[0107] The third image 90 includes a first image area 91 in which an image without the first AI processing is displayed, and a second image area 92 in which an image with the first AI processing is displayed. The first image area 91 is an area, for example, surrounded by a dotted square, set to include a detected specific subject 93 (for example, a "person"). The second image area 92 is an area in the third image 90 other than the first image area 91.

[0108] 17 is a diagram showing an example of recognizing the first image region 91 in the third image 90. The processor 14 performs processing to include information regarding the boundary between the first image region 91 and the second image region 92 in the third image data. For example, the processor 14 performs processing to include information that visually indicates the boundary between the first image region 91 and the second image region 92. The "processing to include information that visually indicates the boundary" refers to image processing that displays the third image 90 based on the third image data, allowing a viewer of the image to determine which areas of the image have not been subjected to the first AI processing.

[0109] 17 , a third image 90 based on the third image data additionally displays a frame 94 of, for example, a specific color indicating the boundary between the first image area 91 and the second image area 92. Note that, in addition to the information about the frame 94, the information about the boundary between the first image area 91 and the second image area 92 may also be, for example, coordinate information indicating the boundary between the first image area 91 and the second image area 92.

[0110] <Image Capture Mode Setting Process> FIG. 18 is a flowchart showing the process of automatically setting the image capture mode in the second embodiment.

[0111] It is assumed that the imaging mode of the imaging device 10 is set to the non-evidence preservation mode (evidence preservation mode=OFF) during image capture (step S11).

[0112] The processor 14 of the imaging device 10 performs, for example, image recognition based on the captured image, and executes a subject detection process to detect a specific subject (such as a person or a flying object) (step S12). The processor 4 determines whether or not a specific subject has been detected in the subject detection process of step S12 (step S13).

[0113] If no specific subject is detected in step S13 (step S13: No), the processor 4 sets the imaging mode of the imaging device 10 to the non-evidence preservation mode (evidence preservation mode=OFF) (step S14), and then returns to step S12 to repeat each process. Note that if the imaging mode has already been set to the non-evidence preservation mode, the non-evidence preservation mode is maintained.

[0114] If a specific subject is detected in step S13 (step S13: Yes), the processor 4 sets the first image area to include the detected specific subject (step S15A). For example, as shown in FIG. 16 above, the processor 4 sets the first image area 91 by surrounding the person with a dotted square. Next, the processor 4 sets the imaging mode of the imaging device 10 to evidence preservation mode (evidence preservation mode = ON) (step S15B), and then returns to step S12 to repeat each process. If the imaging mode is already set to evidence preservation mode, the evidence preservation mode is maintained.

[0115] In this example, the case where the evidence preservation mode is automatically set has been described, but the present invention is not limited to this. For example, when the imaging device 1 detects a specific subject, the imaging device 1 may notify the user of the detection, and the user may switch the evidence preservation mode ON / OFF.

[0116] In the second embodiment, the incidental information indicating whether the first AI processing is executed or not is the same as the example of incidental information shown in the first embodiment (FIGS. 8 and 9).

[0117] As described above, when the evidence preservation mode is set, the imaging device 10 of the second embodiment outputs third image data having a first image area 91 that is an area including an image on which the first AI processing has not been performed and a second image area 92 that is an area including an image on which the first AI processing has been performed, and sets the first image area 91 to include a first subject such as a person or a flying object. According to this configuration, when the imaging mode is set to the evidence preservation mode, the imaging device 1 can output third image data in which an area including a person, a flying object, or the like is set as the first image area 91 that includes an image on which the first AI processing has not been performed, and an area other than the first image area 91 is set as the second image area 92 that includes an image on which the first AI processing has been performed. Therefore, the image of the first image area 91 that has not been subjected to the first AI processing can be recorded in the recording unit 5 as a legally effective evidence image, and the image of the second image area 92 other than the image of the first image area 91 can be displayed on the display unit 6 as an image with good visibility that has been subjected to the first AI processing for confirmation. This allows the image in the second image area 92, which has good visibility, to be more convenient, while leaving the image in the first image area 91, which has high evidential value.

[0118] Furthermore, the imaging device 1 displays a frame 94 indicating the boundary between the first image area 91 and the second image area 92 in the third image 90 based on the third image data. With this configuration, it is possible to easily recognize which area of ​​the image in the third image 90 can be used as an evidence image that has not been subjected to the first AI processing.

[0119] The control method described in the above-described embodiment can be realized by executing a prepared control program on a computer. The control program is recorded on a computer-readable storage medium and executed by being read from the storage medium. The control program may be provided in a form stored on a non-transitory storage medium such as a flash memory, or may be provided via a network such as the Internet. The computer that executes the control program may be included in an information processing device or an imaging device, or may be included in an electronic device such as a smartphone, tablet terminal, or personal computer that can communicate with the information processing device or imaging device, or may be included in a server device that can communicate with these information processing devices, imaging devices, and electronic devices.

[0120] In this embodiment, each process is executed by a computer. The computer may execute these processes by a processor, a program, or a combination thereof. The computer may be a general-purpose computer, a computer for specific applications, a system such as a workstation, or other hardware element capable of executing a program.

[0121] The processor may be configured with one or more pieces of hardware, and the type of hardware is not limited. For example, the processor may be configured with hardware such as a programmable logic device such as a central processing unit (CPU), a micro processing unit (MPU), a field programmable gate array (FPGA), a dedicated circuit for executing specific processing such as an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or a neural processing unit (NPU). The processor also has various units or means for executing various processes in this embodiment. The type of hardware may also be a combination of different types of hardware. When multiple pieces of hardware are configured to execute one or more processes of a certain processor, the multiple pieces of hardware may be located in devices physically separated from each other, or may be located in the same device. Furthermore, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware is configured by an electric circuit (circuitry) or the like that combines circuit elements such as semiconductor elements.

[0122] Furthermore, the present embodiment may be implemented by hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode may be configured by a program. A program may also be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform the respective function. The program may be program code or multiple code segments stored in one or more non-transitory computer-readable media (e.g., storage media or other storages). The program may be stored in multiple non-transitory computer-readable media that reside in physically separate devices. Program code or a code segment may represent a procedure, a function, a subprogram, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. Program code or a code segment may be connected to another code segment or a hardware circuit by sending or receiving information, data, arguments, parameters, or memory contents.

[0123] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the invention.

[0124] This application is based on a Japanese patent application (Patent Application No. 2024-149033) filed on August 30, 2024, the contents of which are incorporated herein by reference.

[0125] REFERENCE SIGNS LIST 1, 10 Imaging device 2, 12 Imaging lens 3, 13 Imaging element 4, 14 Processor 4A, 14A Information processing device 5, 15 Recording unit 6, 16 Display unit 41, 141 Pre-processing unit 42 First image processing unit 43 Second image processing unit 44, 143 Post-processing unit 51, 151 Imaging data 52, 152 Imaging frame 53, 153 Image data 54, 55 Second image data 56, 82 Second image 57, 58 First image data 59, 81 First image 71, 73 Video data 72 Additional information 83, 93 Subject 84, 94 Frame 85 Message image 90, 156 Third image 91 First image area 92 Second image area 142 Image processing unit 154, 155 Third image data

Claims

1. An information processing device having a processor, wherein, when a first mode is set, the processor outputs first image data obtained by not executing a first AI process and second image data obtained by executing the first AI process, and the output destination of the first image data and the output destination of the second image data are different.

2. An information processing device according to claim 1, wherein the first image data is output to a recording unit, and the second image data is output to a display unit.

3. An information processing device according to claim 1, wherein, when the first mode is set, the processor outputs a mixture of the first image data and the second image data.

4. An information processing device according to claim 3, wherein the processor alternately outputs the mixed first image data and second image data.

5. An information processing device according to claim 1, wherein the processor sets the first mode when a first subject is detected.

6. An information processing device according to claim 1, wherein the first AI processing includes at least processing using a trained model.

7. An information processing device according to claim 1, wherein the first image data and the second image data are stream data.

8. An information processing device according to claim 1, wherein, when a second mode different from the first mode is set, the processor outputs the second image data out of the first image data and the second image data.

9. An information processing device according to claim 1, wherein, when the first mode is set, the processor: executes a first processing which is a non-AI processing on the imaging data; outputs the first image data on which the first AI processing has not been executed for the image data on which the first processing has been executed; and outputs the second image data on which the first AI processing has been executed for the image data on which the first processing has been executed.

10. An information processing device according to claim 1, wherein, when the first mode is set, the processor executes a second process, which is a non-AI process, on the first image data and the second image data.

11. An information processing device according to claim 1, wherein the processor generates supplementary information relating to the first AI processing for at least the first image data.

12. An information processing device according to claim 11, wherein the first image data is stream data, and the additional information indicates whether or not the first AI processing has been performed on the first image data over time.

13. An information processing device according to claim 1, wherein the processor performs processing to include, in at least the first image data, information visually indicating whether or not AI processing has been performed.

14. An imaging device comprising an information processing device according to any one of claims 1 to 13.

15. A control method for an information processing device having a processor, wherein, when a first mode is set, the processor outputs first image data obtained by not executing a first AI process and second image data obtained by executing the first AI process, and the output destination of the first image data and the output destination of the second image data are different.

16. A control program for an information processing device having a processor, the control program causing the processor to execute a process to output, when a first mode is set, first image data obtained by not executing a first AI process and second image data obtained by executing the first AI process, and the output destination of the first image data and the output destination of the second image data are different.

17. An information processing device having a processor, wherein when a first mode is set, the processor outputs third image data having a first image area that is a non-execution area of ​​a first AI processing and a second image area that is an execution area of ​​the first AI processing.

18. An information processing device according to claim 17, wherein the processor sets the first mode when a first subject is detected.

19. An information processing device according to claim 18, wherein the first image area includes the first subject.

20. An information processing device according to claim 17, wherein the processor performs processing to include information about the boundary between the first image area and the second image area in the image data.

21. An imaging device comprising an information processing device according to any one of claims 17 to 20.

22. A control method for an information processing device having a processor, wherein, when a first mode is set, the processor outputs image data having a first image area that is a non-execution area of ​​a first AI processing and a second image area that is an execution area of ​​the first AI processing.

23. A control program for an information processing device having a processor, the control program causing the processor to execute a process that, when a first mode is set, outputs image data having a first image area that is a non-execution area of ​​a first AI processing and a second image area that is an execution area of ​​the first AI processing.

Images