Monitoring device, monitoring method, and program

Abstract

It estimates the state of the monitored object with high accuracy. [Solution] The monitoring device (1) includes a control unit (12). The control unit (12) inputs an image to a first machine learning model trained on a dataset labeled with whether or not a monitored object exists, determines whether or not a monitored object is included in the image, and if a monitored object is included in the image, generates a prompt that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. The control unit (12) inputs the code of the image and the code of the prompt to a second machine learning model trained on a dataset labeled with the state of the monitored object, estimates the state of the monitored object in the image.

Description

Technical Field

[0001] The present disclosure relates to a monitoring device, a monitoring method, and a program.

Background Art

[0002] Conventionally, monitoring systems for monitoring video for various purposes have been developed. For example, the monitoring system described in Patent Document 1 detects a predetermined action of a person in a video, sets the person in whom the predetermined action is detected as a detected person, and associates and stores feature information indicating the detected person, the number of detections, information indicating the type of the detected predetermined action, and time information when the predetermined action was performed, weights the number of actions within a predetermined time, sets the detected person as a suspect based on a score, and displays, on a display unit, information that can identify the detected person set as the suspect. Further, this monitoring system further stores the number of detections and identification information for identifying the detected person in association with the feature information indicating the features of the detected person.

[0003] In recent years, with the development of machine learning (deep learning) technology, large language models (LLMs) that understand and process natural language and large vision-language models (VLMs) that understand and process images and natural language have been developed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, LLM / VLM have inferior detection performance compared to recognition-specific models. When using VLM in a surveillance system, it involves overhead images from cameras installed above and estimation from time series data, making it difficult to pinpoint the location of individuals and resulting in poor recognition performance for human behavior.

[0006] In view of these circumstances, the purpose of this disclosure is to provide a monitoring device, a monitoring method, and a program that can estimate the state of a monitored object with high accuracy. [Means for solving the problem]

[0007] A monitoring device according to one embodiment of the present disclosure is a monitoring device comprising a control unit, the control unit inputs an image to a first machine learning model trained on a dataset labeled with respect to whether or not a monitored object exists, determines whether or not the monitored object is included in the image, generates a prompt including coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object if the monitored object is included in the image, and inputs the code of the image and the code of the prompt to a second machine learning model trained on a dataset labeled with respect to the state of the monitored object, estimates the state of the monitored object in the image.

[0008] A monitoring method according to one embodiment of the present disclosure includes the following steps: a monitoring device inputs an image to a first machine learning model trained on a dataset labeled with respect to whether or not a monitored object exists, to determine whether or not the monitored object is included in the image; if the monitored object is included in the image, generates a prompt including coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object; and inputs the code of the image and the code of the prompt to a second machine learning model trained on a dataset labeled with respect to the state of the monitored object, to estimate the state of the monitored object in the image.

[0009] A program according to one embodiment of the present disclosure causes a computer functioning as a monitoring device to input an image into a first machine learning model trained on a dataset labeled with respect to whether or not a monitored object exists, to determine whether or not the monitored object is included in the image; if the monitored object is included in the image, to generate a prompt including coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object; and input the code of the image and the code of the prompt into a second machine learning model trained on a dataset labeled with respect to the state of the monitored object, to estimate the state of the monitored object in the image. [Effects of the Invention]

[0010] According to this disclosure, it becomes possible to estimate the state of the monitored object with high accuracy. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows a schematic configuration of a monitoring system according to one embodiment of the present disclosure. [Figure 2] This is a block diagram showing a first functional example of a control unit of a monitoring device according to one embodiment of the present disclosure. [Figure 3] This figure shows an example of a main image used in a monitoring device according to one embodiment of the present disclosure. [Figure 4] This is a block diagram showing a second functional example of the control unit of a monitoring device according to one embodiment of the present disclosure. [Figure 5] This figure shows an example of a wide-field image used in a monitoring device according to one embodiment of this disclosure. [Figure 6] This flowchart shows an example of operation of a monitoring device according to one embodiment of this disclosure. [Modes for carrying out the invention]

[0012] Hereinafter, one embodiment of the present disclosure will be described in detail with reference to the drawings.

[0013] Referring to Figure 1, the configuration of a monitoring system according to one embodiment will be described. The monitoring system 1 shown in Figure 1 comprises a monitoring device 10 and a camera 20. The monitoring device 10 and the camera 20 are connected to each other so as to be able to communicate via a network 30 including the Internet.

[0014] Camera 20 includes, for example, a CCD (Charge-Coupled Device) camera, a CMOS (Complementary Metal-Oxide-Semiconductor) camera, and a high-speed camera. Camera 20 is installed in vehicles, houses, buildings, shops, streets, etc., and captures (takes) images. Camera 20 may be a security camera, a monitoring camera, a pet camera, etc. Camera 20 may also be a webcam. For example, camera 20 is installed on the ceiling of a vehicle such as an autonomous bus or a passenger car, and captures images of the interior of the vehicle from the ceiling. Camera 20 has a communication function and transmits the captured images to the monitoring device 10 via the network 30.

[0015] The monitoring device 10 is a device that monitors the state of a monitored object in an image. The state of the monitored object includes the object's behavior, actions, posture, presence and direction of movement, and interactions with objects other than the monitored object. The monitored object may be a living organism such as a human or animal, or it may be inanimate. The monitored object may be multiple humans, animals, etc. For example, if the monitored object is a human, the monitoring device 10 monitors the actions of the person shown in the image. The monitoring device 10 comprises a storage unit 11, a control unit 12, and a communication unit 13.

[0016] The storage unit 11 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, RAM (random access memory), ROM (read-only memory), or flash memory. The storage unit 11 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 11 stores information used for the operation of the monitoring device 10 and information obtained by the operation of the monitoring device 10.

[0017] The control unit 12 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or a GPU (graphics processing unit), or a dedicated processor specialized for specific processing. The programmable circuit is, for example, an FPGA (field-programmable gate array). The dedicated circuit is, for example, an ASIC (application specific integrated circuit). The control unit 12 executes processes related to the operation of the monitoring device 10 while controlling each part of the monitoring device 10. Details of the processes of the control unit 12 will be described later.

[0018] The communication unit 13 includes a communication interface for communicating with the camera 20. The communication unit 13 receives an image captured by the camera 20 from the camera 20. The communication unit 13 stores the received image in the storage unit 11.

[0019] <First functional example> Next, two functional examples of the control unit 12 are shown. To distinguish between the two control units 12, the control unit 12 is referred to as the control unit 12a in the first functional example and the control unit 12b in the second functional example. FIG. 2 is a block diagram showing a functional example of the control unit 12a. The control unit 12a shown in FIG. 2 includes a detection unit 121, a prompt generation unit 122, a prompt encoding unit 123, an image encoding unit 124, a concatenation unit 125, and a state estimation unit 126.

[0020] The detection unit 121 acquires an image captured by the camera 20 (hereinafter also simply referred to as "image"). The detection unit 121 inputs the image captured by the camera 20 into the first machine learning model to determine whether the monitoring target is included in the image. The first machine learning model is a trained model trained by a dataset labeled regarding whether the monitoring target exists. The first machine learning model may be stored in the storage unit 11 or may be stored in an external storage device of the monitoring device 10. When the monitoring target is included in the image, the detection unit 121 generates coordinate information indicating the position (coordinates) of the monitoring target in the image and outputs it to the prompt generation unit 122.

[0021] When the monitoring target is included in the image, the prompt generation unit 122 automatically generates a prompt including the coordinate information indicating the position of the monitoring target in the image and the identification information for identifying the monitoring target. The prompt generation unit 122 may generate a prompt including the size of the image. For example, the prompt generation unit 122 generates a prompt such as "At coordinates (x, y) for an X×Y pixel image, a person with ID: abc exists. What is the person with ID: abc doing?" In this example, the "image size" is "X×Y pixels", the coordinate information is "coordinates (x, y)", the monitoring target is "person", and the "identification information" is "ID: abc". Here, by including the image size in the prompt, the second machine learning model described later is more likely to accurately recognize the spatial constraints of the image and perform appropriate processing. In other words, the second machine learning model can utilize the position information more accurately and improve the accuracy of inference.

[0022] The prompt encoding unit 123 encodes (tokenizes and vectorizes) the prompt generated by the prompt generation unit 122 using an arbitrary known encoding method to generate an encoded prompt.

[0023] The image encoding unit 124 acquires the image captured by the camera 20 and encodes (vectorizes) the image using an arbitrary known encoding method to generate an encoded image.

[0024] The concatenation unit 125 concatenates the encoded image and the encoded prompt in association with each other in order to simultaneously input the encoded image generated by the image encoding unit 124 and the encoded prompt generated by the prompt encoding unit 123 to the state estimation unit 126.

[0025] The state estimation unit 126 inputs the encoded image and encoded prompt concatenated by the concatenation unit 125 to the second machine learning model and estimates the state of the monitored object in the image. The second machine learning model is a trained model that has been trained on a dataset labeled with respect to the state of the monitored object. The second machine learning model may be stored in the memory unit 11 or in an external memory device of the monitoring device 10. The "state of the monitored object" may include, if the monitored object is a human, sitting, standing, sleeping, reading, walking, running, lying down, dancing, fighting, playing, etc. Also, if the monitored object is a pet, the "state of the monitored object" may include, lying down, pacing around, jumping, sharpening claws, eating, barking, etc. The state estimation unit 126 outputs the state estimation result to the outside of the monitoring device 10. For example, if camera 20 is a monitoring camera installed on an autonomous bus, the state estimation unit 126 sends the state estimation result to the bus company.

[0026] The image encoding unit 124 may delete at least a portion of the image where the monitored object does not exist and encode the remaining main image. For example, suppose the image captured by the camera 20 is image A shown in Figure 3, and the monitored object is a dog. In this case, the image encoding unit 124 deletes at least a portion of the image where the dog does not exist from image A, and uses the remaining image B as the main image. The image encoding unit 124 then encodes the main image B and outputs it to the concatenation unit 125 (shown by a solid line in Figure 2). Generally, it is considered that deleting the image where the monitored object does not exist improves the accuracy of state estimation. On the other hand, for example, if a person is holding a tool, deleting the tool may lead to an error in state estimation. Therefore, the image encoding unit 124 encodes the entire image and outputs it to the state estimation unit 126 (shown by a dashed line in Figure 2). This allows the state estimation unit 126 to estimate the state of the monitored object by also considering the image where the monitored object does not exist. The image encoding unit 124 may also obtain coordinate information of the monitored object from the detection unit 121. Furthermore, if the detection unit can detect objects related to the monitored object in addition to the monitored object, the image encoding unit 124 may extract and encode the objects related to the monitored object and the monitored object from the image.

[0027] <Example of second function> Figure 4 is a block diagram showing an example of the functions of the control unit 12b. The control unit 12b shown in Figure 4 comprises a detection unit 121, a prompt generation unit 122, a prompt encoding unit 123, an image encoding unit 124, a concatenation unit 125, a state estimation unit 126, an image correction unit 127, and an inverse coordinate correction unit 128. The control unit 12b differs from the control unit 12a in that it further comprises an image correction unit 127 and an inverse coordinate correction unit 128. The other components are the same as those of the control unit 12a, so the same reference numerals are used and explanations are omitted as appropriate.

[0028] The image correction unit 127 corrects the image captured by the camera 20 to generate a corrected image. The image correction unit 127 may perform any known correction process, such as geometric transformation (affine transformation), noise reduction, or edge enhancement. The image correction unit 127 outputs the corrected image to the detection unit 121.

[0029] Figure 5 shows an example of a wide-angle image C captured by the camera 20 when the camera 20 is equipped with a lens with a wide field of view (angle of view) (such as a fisheye lens, ultra-wide-angle lens, or wide-angle lens). If distortion occurs as in the wide-angle image C shown in Figure 5, the image correction unit 127 corrects the distortion.

[0030] The detection unit 121 determines whether or not the monitored object is included in the corrected image generated by the image correction unit 127. If the monitored object is included in the corrected image, the detection unit 121 generates first coordinate information indicating the position (first coordinate) of the monitored object in the corrected image and outputs it to the inverse coordinate correction unit 128.

[0031] The inverse coordinate correction unit 128 performs an inverse correction on the first coordinate, reversing the correction made by the image correction unit 127. For example, if the image correction unit 127 performed an affine transformation, the inverse coordinate correction unit 128 performs an inverse affine transformation. Through this process, the inverse coordinate correction unit 128 can obtain second coordinate information indicating the position of the monitored object (second coordinate) in the image before correction. The inverse coordinate correction unit 128 outputs the second coordinate information to the prompt generation unit 122.

[0032] The prompt generation unit 122 generates a prompt that includes second coordinate information indicating the position of the monitored object in the image before correction, and identification information identifying the monitored object, when the monitored object is included in the corrected image.

[0033] The image encoding unit 124 processes the uncorrected image, similar to the control unit 12a. The state estimation unit 126 estimates the state of the monitored objects in the uncorrected image, similar to the control unit 12a. For example, in the example shown in Figure 5, the state estimation unit 126 estimates the state of monitored objects C1 to C4 using a prompt that includes second coordinate information indicating the respective positions of monitored objects C1 to C4 in the uncorrected image C, and identification information that identifies each of the monitored objects C1 to C4.

[0034] <Operation of monitoring device> Next, an example of the operation of the monitoring device 10 according to one embodiment will be described with reference to Figure 6. In the example shown in Figure 6, the control unit 12 is assumed to have the functions shown in Figure 4.

[0035] In S101, the communication unit 13 receives images captured by the camera 20 via the network 30.

[0036] In S102, the image correction unit 127 corrects the image and generates a corrected image.

[0037] In S103, the detection unit 121 inputs the corrected image into a first machine learning model trained on a dataset labeled with whether or not a monitored object exists, and determines whether or not the monitored object is included in the corrected image. If the monitored object is included in the corrected image, the detection unit 121 proceeds to S104. If the monitored object is not included in the corrected image, the monitoring device 10 does not perform any further processing and returns to S101.

[0038] In S104, the detection unit 121 generates first coordinate information indicating the position (first coordinate) of the monitored object in the corrected image. Subsequently, the inverse coordinate correction unit 128 performs inverse correction on the first coordinate to obtain second coordinate information indicating the position (second coordinate) of the monitored object in the image before correction.

[0039] In S105, the prompt generation unit 122 generates a prompt that includes second coordinate information and identification information that identifies the object to be monitored.

[0040] In S106, the prompt encoding unit 123 encodes the prompt and generates an encoded prompt.

[0041] In S107, the image encoding unit 124 encodes the image and generates an encoded image. This processing by the image encoding unit 124 may be performed between S102 and S106, or in parallel with the processing in S102 to S106.

[0042] In S108, the coupling unit 125 links the encoded image and the encoded prompt together.

[0043] In S109, the state estimation unit 126 inputs the encoded image and the encoded prompt to a second machine learning model, which has been trained on a dataset labeled with respect to the state of the monitored object, and estimates the state of the monitored object in the image.

[0044] As described above, the monitoring device 10 according to this disclosure uses a first machine learning model to determine whether or not a monitored object is included in an image, and if a monitored object is included in the image, generates a prompt that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object, and uses a second machine learning model to estimate the state of the monitored object in the image based on the image and the prompt. According to this disclosure, since the detection result of the monitored object can be reflected in the prompt for the second machine learning model, it is possible to estimate the state of the monitored object with high accuracy. Here, the first machine learning model is specialized in the detection and existence determination of the monitored object, and the second machine learning model is specialized in the detailed estimation of the state, so the processing can be divided and the overall computational load can be distributed in a balanced manner. Furthermore, by adding the position information and identification information of the monitored object when reflecting the detection result of the first machine learning model in the prompt, the estimation accuracy by the second machine learning model can be improved. Furthermore, by using the first machine learning model and the second machine learning model in combination, it is possible to skip the state estimation process when no monitored object is present, thereby reducing unnecessary calculations. By optimizing these processes, high-precision and efficient monitoring can be achieved even in real-time monitoring systems and resource-constrained environments.

[0045] Furthermore, when the monitoring device 10 according to this disclosure corrects an image to generate a corrected image, it uses a first machine learning model to determine whether or not the monitored object is included in the corrected image. If the monitored object is included in the corrected image, it generates a prompt that includes coordinate information indicating the position of the monitored object in the image before correction and identification information identifying the monitored object. Using a second machine learning model, it estimates the state of the monitored object in the image before correction based on the image before correction and the prompt. According to this disclosure, by correcting the image, it becomes possible to estimate the state of the monitored object with high accuracy even in images where state estimation is normally difficult, such as wide-field images. In addition, by performing state estimation on the image before correction, it becomes possible to realize appropriate action estimation based on the situation of the original image.

[0046] <Program> To enable the computer to function as the monitoring device 10 described above, it is also possible to use a computer capable of executing program instructions. The program can cause the computer to perform the operations described above, thereby enabling the computer to function as the monitoring device 10.

[0047] The program can be stored on a non-temporary computer-readable medium. Examples of non-temporary computer-readable media include flash memory, magnetic recording devices, optical discs, magneto-optical recording media, or ROM. The program can be distributed, for example, by selling, transferring, or leasing portable media such as SD (Secure Digital) cards, DVDs (digital versatile discs), or CD-ROMs (compact disc read-only memory) on which the program is stored. The program may also be distributed by storing it in server storage and transferring it from the server to other computers. The program may also be provided as a program product.

[0048] The computer, for example, stores a program stored on a portable medium or a program transferred from a server in the storage unit 11. Then, the computer reads the program stored in the storage unit 11 with the processor (control unit 12) and executes the processing according to the read program with the processor. The computer may also read the program directly from the portable medium and execute the processing according to the program. The computer may also execute the processing according to the received program sequentially each time a program is transferred to the computer from a server. Processing may also be performed by a so-called ASP (application service provider) type service, which does not transfer programs from the server to the computer, but realizes its function only by execution instructions and result acquisition. A program includes information used for processing by an electronic computer that is equivalent to a program. For example, data that is not a direct instruction to the computer but has the nature of defining the computer's processing falls under "information equivalent to a program".

[0049] Although the embodiments described above are representative examples, various modifications or changes are possible without departing from the spirit of this disclosure. For example, multiple component blocks or processing steps described in the embodiments can be combined into one, or one can be divided into multiple.

[0050] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] A monitoring device comprising a control unit, The control unit, A first machine learning model, trained on a dataset labeled with whether or not a target to be monitored exists, is input with an image to determine whether or not the target to be monitored is included in the image. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. A monitoring device that estimates the state of the monitored object in an image by inputting the code of the image and the code of the prompt into a second machine learning model trained on a dataset labeled with respect to the state of the monitored object. [Note 2] A monitoring device comprising a control unit, The control unit, Using the first machine learning model, we determine whether or not the image contains the object to be monitored. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. A monitoring device that uses a second machine learning model to estimate the state of the monitored object in the image based on the image and the prompt. [Note 3] The control unit, The image before correction is corrected to generate a corrected image. Using the first machine learning model, it is determined whether or not the monitored object is included in the corrected image. If the monitored object is included in the corrected image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image before correction, and the identification information. The monitoring device according to Appendix 2, which uses the second machine learning model to estimate the state of the monitored object in the image before correction. [Note 4] The control unit, The aforementioned image is encoded to generate an encoded image, The second prompt is encoded to generate an encoded prompt, A monitoring device according to Appendix 2 or 3, which uses the second machine learning model to estimate the state of the monitored object based on the encoded image and the encoded prompt. [Note 5] The control unit, The monitoring device described in Appendix 4, which removes the portion of the image in which the monitored object does not exist, encodes the remaining image, and generates the encoded image. [Note 6] The subject of the aforementioned surveillance is a person, The aforementioned image is an image taken from the ceiling of the vehicle, of the monitoring device as described in any of Appendix 2 to 5. [Note 7] The prompt includes the size of the image, and is a monitoring device as described in any of appendices 2 to 6. [Note 8] The monitoring device, The process involves inputting an image into a first machine learning model trained on a dataset labeled with whether or not a target being monitored is present, and determining whether or not the image contains the target being monitored. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. The process involves inputting the image code and the prompt code into a second machine learning model trained on a dataset labeled with the state of the monitored object, in order to estimate the state of the monitored object within the image. A monitoring method to perform. [Note 9] The monitoring device, Using the first machine learning model, determine whether or not the image contains the object to be monitored, If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. Using a second machine learning model, estimate the state of the monitored object in the image based on the image and the prompt, A monitoring method to perform. [Note 10] The aforementioned monitoring device The process involves correcting the image before correction to generate a corrected image, Using the first machine learning model, determine whether or not the monitored object is included in the corrected image, If the monitored object is included in the corrected image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image before correction, and the identification information. Using the second machine learning model, estimate the state of the monitored object in the image before correction, The monitoring method described in Appendix 9 is used to perform this task. [Note 11] The aforementioned monitoring device The above image is encoded to generate an encoded image, The process involves encoding the aforementioned prompt to generate an encoded prompt, Using the second machine learning model, estimate the state of the monitored object in the image based on the encoded image and the encoded prompt, The monitoring method described in Appendix 9 or 10, which performs the following: [Note 12] The aforementioned monitoring device The process involves removing the portion of the image in which the monitored object does not exist, encoding the remaining image, and generating the encoded image. The monitoring method described in Appendix 11 is used to perform the following actions. [Note 13] The subject of the aforementioned surveillance is a person, The aforementioned image is an image taken from the ceiling of the vehicle, according to the monitoring method described in any of appendices 9 to 12. [Note 14] The prompt is a monitoring method according to any one of appendices 9 to 13, including the size of the image. [Note 15] A computer that functions as a monitoring device, The process involves inputting an image into a first machine learning model trained on a dataset labeled with whether or not a target being monitored is present, and determining whether or not the image contains the target being monitored. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. The process involves inputting the image code and the prompt code into a second machine learning model trained on a dataset labeled with the state of the monitored object, in order to estimate the state of the monitored object within the image. A program to execute. [Note 16] A computer that functions as a monitoring device, Using the first machine learning model, determine whether or not the image contains the object to be monitored, If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. Using a second machine learning model, estimate the state of the monitored object in the image based on the image and the prompt, A program to execute. [Note 17] To the aforementioned computer, The process involves correcting the image before correction to generate a corrected image, Using the first machine learning model, determine whether or not the monitored object is included in the corrected image, If the monitored object is included in the corrected image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image before correction, and the identification information. Using the second machine learning model, estimate the state of the monitored object in the image before correction, The program described in Appendix 16 for executing this program. [Note 18] To the aforementioned computer, The above image is encoded to generate an encoded image, The process involves encoding the aforementioned prompt to generate an encoded prompt, Using the second machine learning model, estimate the state of the monitored object in the image based on the encoded image and the encoded prompt, A program described in Appendix 16 or 17 for executing the program. [Note 19] To the aforementioned computer, The process involves removing the portion of the image in which the monitored object does not exist, encoding the remaining image, and generating the encoded image. The program described in Appendix 18 for executing this program. [Note 20] The subject of the aforementioned surveillance is a person, The aforementioned image is an image taken from the ceiling of the vehicle, of the program as described in any of appendices 16 to 19. [Explanation of Symbols]

[0051] 1. Monitoring System 10 Monitoring equipment 11 Storage section 12, 12a, 12b Control Unit 13 Communications Department 20 cameras 30 Networks 121 Detection unit 122 Prompt generation unit 123 Prompt Encoding Unit 124 Image Encoding Unit 125 Connection part 126 State Estimation Unit 127 Image Correction Unit 128 Inverse Coordinate Correction Unit

Claims

1. A monitoring device comprising a control unit, The control unit, A first machine learning model, trained on a dataset labeled with whether or not a target to be monitored exists, is input with an image to determine whether or not the image contains the target to be monitored. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. A monitoring device that estimates the state of the monitored object in an image by inputting the code of the image and the code of the prompt into a second machine learning model trained on a dataset labeled with respect to the state of the monitored object.

2. A monitoring device comprising a control unit, The control unit, Using the first machine learning model, we determine whether or not the image contains the object to be monitored. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. A monitoring device that uses a second machine learning model to estimate the state of the monitored object in the image based on the image and the prompt.

3. The control unit, The image before correction is corrected to generate a corrected image. Using the first machine learning model, determine whether or not the monitored object is included in the corrected image. If the monitored object is included in the corrected image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image before correction, and the identification information. The monitoring device according to claim 2, wherein the state of the object being monitored in the image before correction is estimated using the second machine learning model.

4. The control unit, The aforementioned image is encoded to generate an encoded image, The second prompt is encoded to generate an encoded prompt, The monitoring device according to claim 2, wherein the state of the monitored object is estimated based on the encoded image and the encoded prompt using the second machine learning model.

5. The control unit, The monitoring device according to claim 4, which removes the portion of the image in which the object to be monitored does not exist, encodes the remaining image, and generates the encoded image.

6. The subject of the aforementioned surveillance is a person, The monitoring device according to claim 2, wherein the aforementioned image is an image taken from the ceiling of the vehicle showing the interior of the vehicle.

7. The monitoring device according to claim 2, wherein the prompt includes the size of the image.

8. The monitoring device, The process involves inputting an image into a first machine learning model trained on a dataset labeled with whether or not a target being monitored is present, and determining whether or not the image contains the target being monitored. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. The process involves inputting the image code and the prompt code into a second machine learning model trained on a dataset labeled with the state of the monitored object, in order to estimate the state of the monitored object within the image. A monitoring method to perform.

9. The monitoring device, Using the first machine learning model, determine whether or not the image contains the object to be monitored, If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. Using a second machine learning model, estimate the state of the monitored object in the image based on the image and the prompt, A monitoring method to perform.

10. The aforementioned monitoring device The process involves correcting the image before correction to generate a corrected image, Using the first machine learning model, determine whether or not the monitored object is included in the corrected image, If the monitored object is included in the corrected image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image before correction, and the identification information. Using the second machine learning model, estimate the state of the monitored object in the image before correction, The monitoring method according to claim 9, which performs the following:

11. The aforementioned monitoring device The above image is encoded to generate an encoded image, The process involves encoding the aforementioned prompt to generate an encoded prompt, Using the second machine learning model, estimate the state of the monitored object in the image based on the encoded image and the encoded prompt, The monitoring method according to claim 9, which performs the following:

12. The aforementioned monitoring device The process involves removing the portion of the image in which the monitored object does not exist, encoding the remaining image, and generating the encoded image. The monitoring method according to claim 11, which performs the following:

13. The subject of the aforementioned surveillance is a person, The monitoring method according to claim 9, wherein the aforementioned image is an image taken from the ceiling of the vehicle showing the interior of the vehicle.

14. The monitoring method according to claim 9, wherein the prompt includes the size of the image.

15. A computer that functions as a monitoring device, The process involves inputting an image into a first machine learning model trained on a dataset labeled with whether or not a target being monitored is present, and determining whether or not the image contains the target being monitored. If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. The process involves inputting the image code and the prompt code into a second machine learning model trained on a dataset labeled with the state of the monitored object, in order to estimate the state of the monitored object within the image. A program to execute.

16. A computer that functions as a monitoring device, Using the first machine learning model, determine whether or not the image contains the object to be monitored, If the monitored object is included in the image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image and identification information identifying the monitored object. Using a second machine learning model, estimate the state of the monitored object in the image based on the image and the prompt, A program to execute.

17. To the aforementioned computer, The process involves correcting the image before correction to generate a corrected image, Using the first machine learning model, determine whether or not the monitored object is included in the corrected image, If the monitored object is included in the corrected image, a prompt is generated that includes coordinate information indicating the position of the monitored object in the image before correction, and the identification information. Using the second machine learning model, estimate the state of the monitored object in the image before correction, The program according to claim 16 for causing the execution of the program.

18. To the aforementioned computer, The above image is encoded to generate an encoded image, The process involves encoding the aforementioned prompt to generate an encoded prompt, Using the second machine learning model, estimate the state of the monitored object in the image based on the encoded image and the encoded prompt, The program according to claim 16 for causing the execution of the program.

19. To the aforementioned computer, The process involves removing the portion of the image in which the monitored object does not exist, encoding the remaining image, and generating the encoded image. A program according to claim 18 for causing the execution of the following:

20. The subject of the aforementioned surveillance is a person, The program according to claim 16, wherein the aforementioned image is an image taken of the interior of the vehicle from the ceiling of the vehicle.

Images