Monitoring system

Abstract

The invention relates to a monitoring system. A monitoring system according to the present invention may comprise: a camera that captures an image of a monitored area and stores the image; and at least one graphical marker mounted to at least one placement location in the monitoring area, where each of the at least one graphical marker is invisible to the camera when a target is located at a respective placement location, where the camera includes circuitry configured to receive the at least one graphical marker from the camera when the target is located at the respective placement location. The circuitry is configured to capture images of the monitored area at a predetermined frequency, determine whether the graphical indicia are included in each captured image, and send a determination result to the outside of the camera. According to the method, the power consumption of the camera and the dependence of a monitoring system on a network are remarkably reduced, the amount of image / video data needing to be processed by the camera and the complexity of a calculation model are reduced, the speed and accuracy of data transmission are improved, and personal privacy can be safely protected.

Description

Monitoring system

[0001] Citation of Related Applications

[0002] This application claims the benefit of Chinese patent application No. 202210636273.X filed with the State Intellectual Property Office of the People's Republic of China on June 7, 2022, the entire contents of which are hereby incorporated by reference into this document in their entirety. Technical Field

[0003] The present invention relates to a monitoring system comprising an intelligent camera with an image processing function and at least one graphic marker. The monitoring system can monitor target objects in a monitoring area of ​​various application sites. Background Art

[0004] Current surveillance systems typically consist of a camera and an external processing unit that communicates with the camera, such as an edge computer, a cloud processor, or various smart devices like mobile phones. In such a surveillance system, the camera sends real-time images to the processing unit, which then displays and processes the images. At the point of sale, some intelligent surveillance systems can process these images through edge computing and other means to obtain insights into product sales.

[0005] However, these surveillance systems present numerous challenges. First, cameras capture and store large amounts of image / video data and transmit this data to processing units, significantly increasing camera power consumption. Furthermore, data transmission is limited by the bandwidth of wired or wireless networks, resulting in unstable or even poor transmission quality. Second, the data transmitted by cameras may contain images of people, increasing the risk of personal privacy breaches.

[0006] Furthermore, current surveillance systems typically perform image processing within external processing units. For example, in automated surveillance systems such as smart stores / vending machines and smart warehouse monitoring, images sent by cameras are processed using cloud-based processors. This can lead to data transmission delays, loss, and even errors due to poor network transmission quality, resulting in reduced accuracy of image processing results.

[0007] Furthermore, traditional automatic monitoring systems rely on direct detection and analysis of objects in surveillance images / videos. This approach requires large amounts of image data and computation, as well as the creation and training of complex computational models.

[0008] Summary of the Invention

[0009] To solve the above technical problems. The present invention proposes a monitoring system, which includes an intelligent camera with image processing function and at least one graphic marker. The intelligent camera can process the image inside the camera and only send the processing result instead of the complete image to the outside of the camera. In this way, the power consumption of the camera and the dependence of the monitoring system on the network are significantly reduced, and the speed and accuracy of data transmission are improved, and problems such as data packet loss, transmission errors, data delays, etc. of traditional monitoring systems will not occur. At the same time, only sending the processing results as feature data instead of image data can also safely protect personal privacy. In addition, the present invention uses at least one graphic marker to monitor the target object, which greatly reduces the amount of image / video data that needs to be processed, and also significantly reduces the complexity of the calculation model.

[0010] A monitoring system according to the present invention may include: a camera that captures images of a monitoring area and stores the images; and at least one graphic marker mounted to at least one storage location in the monitoring area, wherein each of the at least one graphic marker is invisible to the camera when an object is located at the corresponding storage location, wherein the camera includes a circuit configured to: capture images of the monitoring area at a predetermined frequency, determine whether the graphic marker is included in each captured image, and transmit the determination result to an exterior of the camera.

[0011] Preferably, the circuit of the camera is further configured to extract target object information from the image of the monitoring area and transmit the target object information to the outside of the camera.

[0012] Preferably, each of the at least one graphic marking has a selected color that is different from the color of the object to be placed on the corresponding placement location.

[0013] According to the monitoring system of the present invention, a plurality of graphic marks can be respectively installed at a plurality of placement positions in the monitoring area, and the plurality of graphic marks are different from each other. For example, the plurality of graphic marks can have different patterns, shapes and / or colors.

[0014] Preferably, the circuit of the camera is further configured to associate each image mark with a corresponding placement location and / or an object to be placed at the corresponding placement location.

[0015] Preferably, the circuit of the camera is further configured to adjust the shooting parameters of the camera according to the recognition accuracy of the at least one graphic mark in the multiple images of the monitoring area by capturing and analyzing the multiple images of the monitoring area.

[0016] Preferably, in order to determine and / or inspect the target object, the circuit of the camera is also configured to: intercept an area of ​​interest from the first image of the monitoring area, the area of ​​interest including the target object of interest located at the corresponding placement position, capture a second image of the area of ​​interest, and extract the target object information of the target object of interest from the second image.

[0017] Similarly, in order to determine the graphic mark, preferably, the circuit of the camera is also configured to: intercept a region of interest from the first image of the monitoring area, the region of interest containing the graphic mark of interest among the at least one graphic mark, capture a second image of the region of interest, and determine whether the graphic mark of interest is included in the second image.

[0018] Preferably, the camera circuit is further configured to receive an external signal, and when the external signal indicates that a person has entered the monitoring area, the camera begins capturing images of the monitoring area at the predetermined frequency. Optionally, the camera circuit is further configured to detect whether a person has entered the monitoring area by detecting and extracting facial features from images of the monitoring area.

[0019] Preferably, the camera circuit is further configured to transmit the determination result and a fill signal indicating an empty storage location when it is determined that the graphic mark is included in one captured image. Also preferably, the camera circuit is further configured to transmit the determination result and a fill signal indicating an empty storage location when it is determined that the graphic mark is included in a predetermined number of consecutive captured images.

[0020] The camera of the monitoring system according to the present invention comprises an integrated chip on which the circuit of the camera is arranged, and a memory is further included in the integrated chip, wherein the memory stores a predetermined algorithm for image processing of the circuit.

[0021] Detailed and preferred embodiments of the present invention will be described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a schematic diagram showing a first embodiment of a monitoring system according to the present invention.

[0023] FIG. 2 is a schematic diagram showing one embodiment of a graphic mark of a monitoring system according to the present invention.

[0024] FIG. 3 is a schematic diagram showing an embodiment of a chip structure of a camera of a monitoring system according to the present invention.

[0025] FIG4 is a schematic diagram showing a second embodiment of a monitoring system according to the present invention.

[0026] FIG5 is a schematic diagram showing a third embodiment of a monitoring system according to the present invention.

[0027] FIG6 is a schematic diagram showing an embodiment of a monitoring system camera recognizing and extracting facial features of a human face according to the present invention.

[0028] FIG. 7 is a schematic diagram illustrating an embodiment of an image captured by a camera of a monitoring system according to the present invention.

[0029] FIG8 is a schematic diagram showing an embodiment in which a camera of a monitoring system according to the present invention determines a graphic mark in a monitoring area.

[0030] FIG9 is a schematic diagram showing another embodiment of determining a graphic mark in a monitoring area by a camera of a monitoring system according to the present invention.

[0031] FIG. 10 is a schematic diagram illustrating a flow chart of an embodiment of a method in which a camera of a monitoring system determines whether a graphic mark exists in an image according to the present invention.

[0032] FIG. 11 is a schematic diagram illustrating a workflow of a camera of a monitoring system according to the present invention. DETAILED DESCRIPTION

[0033] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Unless otherwise specified, the same reference numerals represent the same or equivalent components.

[0034] Figure 1 illustrates a first embodiment of a monitoring system according to the present invention. As shown in Figure 1 , a camera 1 of the monitoring system is installed within a monitoring area, enabling panoramic surveillance of the monitoring area. Multiple objects 2 are arranged within the monitoring area, each placed in a corresponding storage location 3. A supply source 8 of the objects is also located within the monitoring area, ensuring timely replenishment of the objects 2 to the corresponding storage locations 3 when needed. For example, the monitoring area may be a vending machine, with the objects 2 being merchandise. Based on real-time monitoring by the camera 1, upon receiving a refill instruction, the vending machine automatically refills the corresponding storage locations with the objects 2 stored in the supply source 8. Alternatively, the monitoring area may be a storage area such as a warehouse, with the objects 2 being stored goods and the storage locations 3 being inventory locations within the warehouse. Based on real-time monitoring by the camera 1, upon receiving a refill instruction, the objects 2 from the supply source 8 are automatically refilled to the corresponding storage locations 3. Of course, the monitoring area is not limited to this embodiment; the monitoring system according to the present invention can be applied to any location requiring monitoring of target objects.

[0035] The monitoring system also includes at least one graphic marker 4. In the embodiment shown in Figure 2, multiple graphic markers 4 are mounted on corresponding target object locations 3. For example, the graphic markers 4 can be adhered to the target object locations 3 using an adhesive, without requiring any modification to the target object locations 3. Alternatively, the graphic markers can be mechanically embedded into the corresponding target object locations 3. This approach reliably prevents the graphic markers from falling or becoming dislodged due to operational errors.

[0036] As shown in the upper image of FIG. 2 , when the target 2 is placed in the corresponding storage location 3 according to a predetermined layout, the graphic mark 4 is invisible to the camera 1. For example, the graphic mark 4 can be adhered to the surface of the storage location 3 so that the target 2 will block the graphic mark 4 after it is placed. As shown in the lower image of FIG. 2 , when the target 2 is not in the corresponding storage location 3, the graphic mark 4 on the storage location 3 is exposed and can then be captured by the camera 1. Based on the detection of the graphic mark 4 in the captured image, the camera 1 can determine that the corresponding storage location is empty and then send the judgment result to the outside of the camera, such as an external processing unit. Based on the received judgment result of the existence of an empty storage location (empty storage location 3), the external processing unit sends an instruction to the monitoring area, and the monitoring area executes corresponding measures according to the instruction, such as filling the empty storage location or issuing an alarm signal.

[0037] Figure 2 shows a plurality of objects 2 and a plurality of graphical markers 4 placed in a plurality of object locations 3. Each graphical marker 4 is distinct, so that when the camera 1 recognizes a graphical marker 4, it can simultaneously determine the specific location of the corresponding location and the corresponding object information (e.g., type / category, etc.). In this regard, the circuitry included in the camera 1 can pre-associate each graphical marker 4 with the corresponding location 3 and / or object 2 type, location coordinates, and other information.

[0038] Multiple graphic marks can be distinguished by their graphics, shapes, and / or colors. Of course, the present invention is not limited to these. Any graphic marks that can be distinguished from one another and thus distinguished by a camera in an image are applicable to the present invention. At the same time, each graphic mark should also be distinguishable from its corresponding target object, allowing the camera to accurately identify the graphic mark and target object separately. For example, the image mark may include one or more colors that are different from the colors contained in the target object. Specifically, as shown in the lower image of Figure 2, graphic mark 4 is lighter in color (white in the figure) and has diagonal stripes compared to target object 2. By combining the graphics with the recognition of these different colors, camera 1 can easily and quickly detect the graphic mark in the captured image, thereby determining an empty storage location. For ease of illustration and understanding, Figure 2 only shows graphic marks and targets distinguished by a single color shade and a simple stripe pattern. It is easy to understand that graphic marks can be composed in a variety of ways, including but not limited to a combination of one or more of graphics, shapes, colors, special symbols, etc.

[0039] The following is a detailed description of camera 1 used in the surveillance system of the present invention. Camera 1 is an intelligent camera with built-in image processing capabilities. Camera 1 internally includes an integrated circuit chip, which may have a stacked structure as shown in Figure 3, such as a stacked complementary metal oxide semiconductor (CMOS) image sensor chip. Specifically, the integrated chip is constructed from a stacked substrate structure. A pixel array, consisting of multiple pixels, is arranged on a first substrate in the stack. The pixel array converts light signals received by the camera lens into electrical signals through photoelectric conversion and transmits these electrical signals to a second substrate in the stack. Memory and processing circuitry are arranged on the second substrate, which is bonded to the first substrate (the bonding structure is not shown) for mutual communication. Optionally, the memory may also be located on a third substrate (not shown) in the stack. That is, the pixel array, memory, and processing circuitry may be located on separate substrates. It should be understood that the stacked arrangement of multiple substrates described above is merely exemplary. Components such as the pixel array, pixel circuitry, processing circuitry, and memory may be designed and stacked on multiple substrates as needed. The processing circuitry, for example, includes a digital signal processor (DSP), which generates image data based on electrical signals and stores the image data in memory. The memory also stores predetermined algorithmic models, and the processing circuitry can perform various image processing operations based on these algorithmic models, such as detecting the presence of graphic markers in surveillance images, as well as various image processing operations described in detail below.

[0040] As a result, the monitoring system's image processing can be performed on the camera itself, rather than on an external processor (edge ​​computer, cloud server, mobile phone terminal, etc.) connected to the camera by wire or wirelessly. Of course, the camera 1 can also access the network, for example, to obtain or update algorithm models.

[0041] An example of image processing performed by camera 1 is shown in Figure 8. Figure 8 shows an example image of the surveillance area captured by camera 1. The left image in Figure 8 shows the first image captured by camera 1 at a specific moment. After processing the first image, the circuitry in camera 1 preliminarily detects graphical markers 4 and 4'. Further computational processing, such as comparison with pre-stored images of standard graphical markers, confirms that graphical marker 4 has been detected (e.g., with a probability greater than 90%). Simultaneously, the circuitry determines that the probability of graphical marker 4' being graphical marker 4 is less than 90%. At this point, the circuitry extracts the region of interest containing graphical marker 4' from the first image (as indicated by the dashed box in the left image of Figure 8) and instructs camera 1 to adjust capture parameters such as focal length, resolution, exposure time, and possibly other parameters (white balance, noise reduction, etc.). The camera then captures the region of interest containing graphical marker 4', obtaining a higher-quality second image, as shown in the right image of Figure 8. The circuitry processes the clear second image to confirm detection of graphical marker 4 and then transmits the result to the camera's external environment.

[0042] The camera circuitry can transmit only the detection result externally. For example, in the embodiment shown in Figure 8, only the detection of two graphical markers 4 is transmitted as feature data to the external processing unit. Based on this detection result, the external processing unit determines that two vacant locations exist in the monitoring area and obtains the location information and target object information associated with these two graphical markers 4. The processing unit can then issue corresponding instructions to the monitoring area, such as refilling the target object 2 to the corresponding vacant location and / or causing the monitoring area to issue an alarm.

[0043] Compared to traditional surveillance systems that require sending large amounts of image / video data to external processing units for processing and interpretation, the surveillance camera of the present invention only transmits detection results (e.g., "Graphic mark 4 is found at positions m and n where object 2 is placed," or "Positions m and n are vacant") as feature data. This significantly reduces the amount of data transmitted, thereby significantly improving transmission quality and speed. Furthermore, traditional surveillance systems monitor objects by detecting and interpreting the image of the target within surveillance images, which requires complex modeling and training. In contrast, the present invention uses graphical markers to indirectly monitor objects by detecting them within surveillance images. The use of simple graphical markers greatly simplifies the computational modeling and training process, significantly reduces the amount of data required for processing, and thus significantly improves image processing speed. Furthermore, compared to traditional surveillance systems, the surveillance camera of the present invention, due to its inherent processing capabilities, can automatically and in real time adjust its shooting parameters and angles as needed. Only one camera is required per surveillance area.

[0044] Optionally, in certain circumstances, the camera can also transmit partial image data externally. For example, in certain special cases, the camera may be unable to accurately determine whether a graphical marker has been detected (e.g., after a predetermined number of repeated captures and detections, the probability remains below 90%). In such cases, the camera can, upon request from an external processing unit, transmit the second image (as shown in FIG. 8 ) and / or an image of the area of ​​interest captured from the first or second image to the external camera for evaluation by the user or a more complex recognition system. In this case, the amount of image data required to be transmitted is still significantly less than that required by traditional surveillance systems, as these systems do not perform any processing on the captured images. Alternatively, and as will be readily understood, the camera can also transmit the complete image data externally. In this case, the external processing unit can issue a request for the complete data to the camera.

[0045] In the manner described above, the camera 1 can automatically, quickly and accurately determine whether the target object exists in the monitoring area (for example, whether it is sold, unloaded, stolen or accidentally dropped, etc.) using the graphic mark 4.

[0046] In a similar manner, camera 1 can also determine whether the layout of the objects is correct. Specifically, camera 1 captures a first image of the monitored area. The camera circuitry extracts a region of interest from the first image, which contains at least one object. The camera circuitry processes the region of interest to obtain object information. For example, the camera circuitry can identify the type / category of the object by comparing and matching the object in the image with multiple pre-stored images of objects. If the object is a commodity with a machine-readable barcode, such as a QR code or barcode, the camera circuitry can also obtain information such as the type / class of the object by identifying the QR code or barcode on the object's packaging. Furthermore, similar to the method shown in FIG8 , if the camera circuitry cannot accurately obtain object information from the first image, it can adjust the capture parameters to capture a second image containing the region of interest, and then extract object information from the second image with improved image quality. The camera circuitry can then compare the object information with pre-stored layout information to determine whether the object is correctly positioned. Optionally, the camera circuitry can also transmit the object information to an external processing unit for subsequent processing and determination.

[0047] Similarly to the processing method in the embodiment of FIG8 , the camera circuit can only transmit the target object information data (e.g., "X / X Coke") extracted and recognized from the image, rather than the large amount of image data, thereby significantly reducing the transmission pressure on the network bandwidth. Of course, the camera circuit can also transmit partial or complete image data to the external device based on external requests.

[0048] In addition, when the camera is installed in the monitoring area, the camera circuit can pre-adjust and set the shooting parameters based on the recognition accuracy of the graphic marks in the monitoring area image, for different monitoring areas, the targets therein, and the layout of the targets. Specifically, the camera can pre-shoot the monitoring area multiple times with multiple sets of different shooting parameters, and process and analyze the multiple sets of images obtained. The shooting parameters here include, but are not limited to, shooting angle, resolution, exposure time, white balance, noise reduction, etc. Based on the shooting parameters of the image frames with higher recognition accuracy of the graphic marks in the multiple sets of images, the camera circuit determines the preferred shooting parameter range suitable for the monitoring area and its targets. Therefore, based on the determined shooting parameter range, the camera circuit can pre-set the shooting parameters for the monitoring area. In this way, the judgment efficiency and accuracy during monitoring are further improved.

[0049] FIG4 illustrates a second embodiment of a monitoring system according to the present invention. In this second embodiment, the monitoring system is installed in a surveillance area, such as a vending machine store. As shown in FIG4 , a camera 1 is positioned so that its field of view covers a sales area containing multiple objects 2 and a stock area 5. The sales area includes multiple storage locations (not shown), each of which is provided with a graphical marker (not shown). The graphical markers are similar to those described above with respect to FIG2 . The surveillance area is also provided with a sensor (not shown) that detects whether a person has entered the surveillance area. Camera 1 is connected to the sensor and receives signals from the sensor. When the sensor signals that a person has entered the surveillance area, camera 1 exits standby mode, begins capturing images of the surveillance area at a predetermined frequency, and processes the images. Upon detecting one or more graphical markers, camera 1 transmits the detection results to an external processing unit. Optionally, based on the detection results, camera 1 directly transmits a fill signal to the stock area 5, instructing the stock area 5 to fill the corresponding empty locations with objects. After determining that the empty locations are filled, camera 1 captures the inventory area 5 and processes the captured images to determine whether the supply source of the objects is empty. If it is determined that there is an empty supply source, the determination result is sent to the external processing unit. After receiving the reminder from the external processing unit, the user can replenish the supply source accordingly.

[0050] FIG5 illustrates a third embodiment of a surveillance system according to the present invention. This third embodiment differs from the second embodiment in that the surveillance system's cameras 1 are arranged to simultaneously monitor the sales area, the storage area 5, and the entrance to the monitored area. In other words, the cameras 1 also monitor whether anyone enters the monitored area.

[0051] Specifically, as shown in Figure 5, the camera captures the entrance to the surveillance area at a predetermined frequency and determines whether someone has entered the surveillance area by detecting and extracting facial features of the face in the image (i.e., the dot-shaped facial feature map in Figure 6). Detecting facial features generally involves locating the face in the image and detecting key facial structural features. First, the camera circuit uses an algorithm based on deep learning, for example, to locate the face in the image and obtain facial boundary points. Then, key facial structural features in the facial area, such as the mouth, right eyebrow, left eyebrow, right eye, left eye, nose, and jaw, are detected within the facial boundary to obtain the feature points shown in Figure 6. It should be understood that any known facial feature detection and extraction method can be selected as needed, and the relevant algorithm can be stored in the camera's chip. Of course, when the camera is connected to the network, relevant calculations can also be performed and the calculation results obtained through the cloud.

[0052] The camera circuit determines that someone has entered the monitoring area based on the facial feature data detected, and based on this determination, begins to photograph the sales area at a predetermined frequency. Optionally or preferably, the camera circuit can send the facial feature data to the outside. For example, as shown in FIG6 , the camera 1 sends the detected facial feature map to the processing unit 7 outside the camera via the network 6. The processing unit 7 and the external processing unit mentioned in this article can be a remote computer as shown in FIG6 , or a cloud processor, an application, or an application on a mobile phone terminal, etc. As shown in FIG6 , what is transmitted between the camera 1 and the processing unit 7 via the network 6 is the facial feature map extracted by the camera 1. This data greatly reduces the amount of data transmission and the transmission speed compared to both the original image data and the facial image data. In this way, the need to transmit images with a huge amount of data is avoided, and the exposure of consumers' privacy can be avoided.

[0053] Furthermore, as shown in the left image of Figure 7 , camera 1 may capture an image containing a human hand. In certain applications, camera 1 may process the human hand in the image, extracting visual markers such as birthmarks or tattoos that may be present on the hand, and transmit these visual markers to an external device. This visual marker can then be used by an external processing unit to assist in identifying the target subject possessing the hand. Because camera 1 processes the image and only transmits the visual markers to the external device, the efficiency of the external processing unit in identifying the target subject can be improved. Conversely, in other applications, when the target subject is a person whose private information requires protection (for example, when the monitored area is a sales location and the target subject is an ordinary consumer), if camera 1 detects a human hand, it will not transmit the image containing the hand to the external device. In this way, by configuring camera 1 differently for different monitoring locations / situations, different objectives can be achieved.

[0054] Furthermore, the right image in Figure 7 shows an image captured by camera 1 containing graphic mark 4. As indicated by the arrows in Figure 7 , if the left and right images appear sequentially in consecutive image frames (e.g., at 15 fps), camera 1 determines that the corresponding target object 2 has been removed or sold. Conversely, if the image shown in Figure 9A and the right image in Figure 7 appear sequentially in consecutive image frames, camera 1 determines that target object 2 may not have been removed but may have accidentally fallen. In this way, detecting a human hand in an image can also assist camera 1 in determining the condition of merchandise or goods within the monitored area.

[0055] Furthermore, when the camera captures the image on the left side of Figure 7, it may be unable to accurately determine whether graphical marker 4 is present due to a hand obstructing the view. In this case, the camera will wait for a predetermined period of time (for the hand to leave) before retaking the image. If the camera is still unable to determine the presence of a suspicious person after the predetermined period, for example, if the suspicious person remains within the monitored area, the camera circuitry will issue a notification signal to alert the user.

[0056] Figure 9 illustrates an embodiment in which camera 1 detects a graphical marker 4. Figure 9A shows an image captured by camera 1 while monitoring a surveillance area. If no graphical marker is detected in this image, camera 1 does not transmit any signals or images to the external device. Camera 1 continues operating, and when it captures the image shown in Figure 9B , the camera circuit detects the graphical marker 4 and then transmits only the detection result to the external device. Alternatively, the camera circuit transmits the detection result along with a fill signal indicating an empty storage location. Alternatively, the camera circuit transmits the fill signal directly to supply source 8 as shown in Figure 1 or to storage area 5 as shown in Figures 4 and 5 . Upon receiving the fill signal from the external device or the camera circuit, supply source 8 or storage area 5 replenishes objects 2 to the corresponding storage locations. Figure 9C shows an image captured by camera 1 of objects 2 being filled. Based on this image and the subsequent image of completed replenishment (as shown in Figure 9A ), camera 1 determines that replenishment has been successful. If, within a predetermined period of time after the fill signal is issued, the camera captures the image shown in Figure 9D , replenishment is determined to have failed and this determination is transmitted to the external device. Simultaneously, camera 1 photographs supply source 8 or storage area 5, analyzes the image, determines whether the supply source 8 or the corresponding target item in storage area 5 is empty, and transmits the determination result externally. If camera 1 determines that the supply source of the corresponding target item is empty, it can optionally simultaneously transmit a signal to the external device to refill the supply source. On the other hand, if camera 1 determines that the supply source of the corresponding target item is not empty, this could be due to a malfunction in the automatic filling device or a jamming of the target item to be filled. In this case, the circuitry of camera 1 transmits a warning signal externally, prompting the external processing unit to alert the user to inspect and repair the supply source.

[0057] It's worth noting that even though the graphical marker 4 is partially exposed and captured and detected by camera 1 at the moment shown in FIG9C, camera 1 does not inappropriately issue a fill signal in such a situation. Based on the algorithm stored in camera 1, the camera only issues a determination result indicating an empty level (and optionally, a fill signal) when it detects the complete graphical marker 4 shown in FIG9B. Accordingly, even if the image shown in FIG9C is captured outside the sequence shown in FIG9A-B, for example, while object 2 is being removed, camera 1 will not erroneously issue a determination result indicating an empty level.

[0058] Figure 10 illustrates a specific example of how the camera circuit of a surveillance system according to the present invention determines whether a graphic mark exists in an image. As shown in Figure 10 , when the camera circuit processes a captured image, if the probability of detecting a graphic mark is determined to be 90% or greater based on the algorithm model stored in the camera, the graphic mark is determined to be detected. If the probability of detecting a graphic mark is determined to be 30% or less, the graphic mark is determined not to be detected.

[0059] On the other hand, if the probability of detecting the graphic mark is determined to be greater than 30% and less than 90%, the camera circuit can adjust the shooting parameters and shoot again (count + = 1) to analyze and judge again based on the optimized image until it can determine whether the graphic mark is detected. This step is repeated up to 3 times. If it is determined that the graphic mark is detected (i.e., the probability is 90% or more) during the xth (x≤3, i.e., count<=3 as shown in Figure 10) shooting and analysis, it is determined that the graphic mark is detected, that is, the probability is 90% or more, then the process proceeds to step S5 as shown in Figure 11; if it is determined that the graphic mark is not detected (i.e., the probability is 30% or less) during the xth (x≤3) shooting and analysis, then the process proceeds to step S4 as shown in Figure 11.

[0060] On the other hand, if after three repetitions, the camera circuit still cannot confirm whether the graphic mark has been detected (with the probability falling between 30% and 90%) (i.e., count > 3 as shown in Figure 10), the camera circuit issues a warning signal to the outside world. After receiving the warning, the user can request an image of the monitored area from the camera through an external processing unit. The user can then make a manual judgment based on the partial / complete image of the monitoring area sent by the camera. Alternatively, if the camera circuit cannot determine due to a malfunction or obstruction, the user can visit the monitored area for inspection.

[0061] It should be understood that the probability values ​​of 90% and 30% shown in FIG10 are merely exemplary, and the user may select other values ​​as thresholds based on design requirements, usage scenarios, etc., such as 80% and 20%.

[0062] FIG11 illustrates an example of the operating steps of a camera in a monitoring system according to the present invention. As shown in FIG11 , in step S1, a determination is made as to whether a person has entered the monitoring area. For example, in the embodiment shown in FIG4 , a sensor located at the entrance to the monitoring area sends a signal to camera 1 when detecting that a person has entered the monitoring area. If the signal is not received, camera 1 enters a standby state. Alternatively, as in the embodiment shown in FIG5 , when camera 1 is arranged so that its viewing angle covers the entrance to the monitoring area, the entrance to the monitoring area can be photographed, and the image of the entrance to the monitoring area can be processed to determine whether a person has entered the monitoring area. The determination method here is the same as that described above based on FIG6 .

[0063] After the camera 1 detects that someone has entered the monitoring area or receives an external signal that someone has entered the monitoring area, the camera 1 starts to shoot the monitoring area, as shown in step S2. In S2, the camera 1 shoots the monitoring area at a predetermined frequency. The camera circuit enters step S3 when it is possible that a graphic mark 4 is detected in the captured image. In S3, the camera circuit determines whether there is an empty position by determining whether a graphic mark is detected. When it is determined that the graphic mark 4 is not detected in a currently captured image, or preferably, when the graphic mark 4 is not detected in multiple frames (a pre-set number) of images captured continuously, the camera circuit makes a judgment that there is no empty position and then returns to S1. On the other hand, in S3, when the camera circuit determines that a graphic mark 4 exists in a currently captured image, it judges that the storage position corresponding to the graphic mark 4 is empty. Optionally or preferably, for example, to improve judgment accuracy, when the camera circuit detects a graphical marker 4 in multiple (predetermined) consecutive frames of imagery, it determines that the storage location corresponding to that graphical marker 4 is vacant and obtains information about the storage location and / or target object associated with that vacant location. When multiple graphical markers 4 are detected simultaneously, the camera circuit separately determines information about the storage location (e.g., its specific location within the monitoring area) and / or information about the target object 2 (e.g., its type / class) corresponding to each graphical marker 4. Furthermore, in step S3, the camera 1 circuit may detect that the monitoring area is blocked by an obstacle, such as a consumer or other person wandering in front of the sales area, or a consumer or other person's hand or other obstructing a portion of the camera 1's field of view. In this case, as shown in FIG10 , if the camera still cannot determine whether an object or graphical marker exists in the captured image after repeated detection a predetermined number of times (e.g., three times), the camera circuit will issue a warning signal to the user, prompting them to address the obstacle / obstruction. Optionally, upon user request, the camera can also transmit a portion of the captured image to the external device for manual judgment by the user.

[0064] When one or more graphic marks 4 are detected in S3, step S5 is entered. In S5, the camera circuit sends the determination result in S3 and the acquired characteristic information (for example, "the storage position at a certain row and column for placing target object X is empty", "a certain disposal position for placing target object X, a certain disposal position for placing target object Y... are empty") to the outside instead of sending the captured image to the outside. Optionally, the camera circuit sends a filling signal to the outside together with the above determination result. After receiving the above determination result (and optionally, the filling signal), the external processing unit issues an instruction to instruct the monitoring area to fill the target object into the corresponding empty position. Alternatively, the camera circuit can directly send the filling signal to the supply source of the monitoring area (for example, the supply source 8 shown in Figure 1 or the inventory area 5 shown in Figure 4), instructing the supply source 8 or the inventory area 5 to fill the corresponding empty position with the corresponding target object. Then, step S6 is entered. In S6, wait for a predetermined time (for example, 2 seconds) to complete the target object filling. Then step S7 is entered. In S7, the camera circuit detects whether the empty position is actually filled. In this step, similar to S3, it is determined whether a graphic mark is detected in the captured image. If the graphic mark 4 is not detected, the camera circuit determines that the empty position has been filled and then returns to S1; if the graphic mark 4 is detected, it proceeds to step S8. In S8, it is determined whether the corresponding target object stored in the supply source 8 or the inventory area 5 is empty. Specifically, the camera 1 photographs the supply source 8 or the inventory area 5, and by processing the photographed image, detects whether the supply source 8 or the inventory area 5 of the corresponding target object is empty. If it is determined that the supply source of the corresponding target object is empty, it proceeds to step S10. In S10, the camera circuit sends the judgment result to the outside. Optionally, the camera circuit sends the instruction to replenish the supply source together with the judgment result to the outside. After receiving the signal to replenish the supply source from the external processing unit, the user replenishes the supply source 8 or the inventory area 5 accordingly.

[0065] In S7, when the camera circuit cannot determine whether the empty position is filled (for example, the captured image is not clear, there are obstacles blocking it, etc.), the process goes to step S9. In S9, the camera circuit optimizes the shooting parameters, and repeats the shooting and judgment until it can accurately determine whether the empty position is filled. Similar to the embodiment shown in Figure 10, the steps can be repeated up to 3 times. If it is possible to determine whether the graphic mark is detected during the xth (x≤3) shooting and analysis (so that it can be determined whether the empty position is filled), the process goes to S7; if it is still impossible to determine whether the empty position is filled after repeating 3 times (that is, x>3), the camera circuit sends a reminder to the outside. The user can go to the monitoring area to check, for example, to clear obstacles.

[0066] The above-described processing steps are stored as a program in the memory of the integrated chip of camera 1. This program is used to instruct the camera to perform corresponding operations during the monitoring process when the processing circuitry within the camera reads the program. It will be appreciated that the above-described program can also be stored in an external processing unit or computer-readable storage medium. In exceptional circumstances (e.g., when the camera circuitry is unable to process images normally), the external processing unit / computer can retrieve the images captured by the camera and process them externally.

[0067] The image processing mentioned above, including the processing of graphic marks, objects, and facial features in the image, can be performed using known algorithms for image recognition or modeling using known computational methods. Detailed descriptions are omitted here.

[0068] Although the above embodiments are described by taking the monitoring system deployed in a vending machine / store as an example, the present invention is obviously not limited thereto, but can be applied to any occasion where there is a need to monitor a target object.

Claims

1. A monitoring system comprising: a camera that captures images of the surveillance area and stores the images; as well as at least one graphic mark mounted to at least one storage location in the monitoring area, wherein each of the at least one graphic mark is not visible to the camera when an object is located at the corresponding storage location, The camera includes a circuit configured to: capturing images of the monitoring area at a predetermined frequency, determining whether the graphical mark is included in each captured image, and The determination result is transmitted to the outside of the camera.

2. The monitoring system according to claim 1, wherein: The circuit is further configured to extract target object information from the image of the monitoring area and transmit the target object information to an outside of the camera.

3. The monitoring system according to claim 1, wherein: Each of the at least one graphic mark has a selected color that is different from a color of the object to be placed on the corresponding placement location.

4. The monitoring system according to claim 1, wherein: A plurality of graphic marks are respectively installed at a plurality of placement locations in the monitoring area, and the plurality of graphic marks are different from each other.

5. The monitoring system according to claim 4, wherein: The plurality of graphic marks have different patterns, shapes and / or colors.

6. The monitoring system according to claim 4, wherein: The circuit is further configured to associate each of the image marks with the corresponding placement location and / or the object to be placed at the corresponding placement location.

7. The monitoring system according to claim 1, wherein: The circuit is further configured to capture and analyze multiple images of the monitoring area and adjust the shooting parameters of the camera according to the recognition accuracy of the at least one graphic mark in the multiple images of the monitoring area.

8. The monitoring system according to claim 2, wherein: The circuit is further configured to: A region of interest is captured from the first image of the monitoring area, wherein the region of interest includes a target object of interest located at a corresponding placement position. capturing a second image of the region of interest, and The target object information of the target object of interest is extracted from the second image.

9. The monitoring system according to claim 1, wherein: The circuit is further configured to: intercepting a region of interest from the first image of the monitoring area, wherein the region of interest includes a graphic mark of interest in the at least one graphic mark; capturing a second image of the region of interest, and It is determined whether the graphical mark of interest is included in the second image.

10. The monitoring system according to claim 1, wherein: The circuit is further configured to receive an external signal, and when the external signal indicates that someone enters the monitoring area, the camera starts to capture images of the monitoring area at the predetermined frequency, or, The circuit is further configured to detect whether a person enters the monitoring area by detecting and extracting facial features of a person from an image of the monitoring area.

11. The monitoring system according to claim 1, wherein: The circuit is further configured to transmit a determination result and a fill signal indicating an empty storage location when it is determined that the graphic mark is included in one captured image.

12. The monitoring system according to claim 1, wherein: The circuit is further configured to transmit a determination result and a filling signal indicating an empty storage location when it is determined that the graphic mark is included in a predetermined number of consecutive captured images.

13. The monitoring system according to any one of claims 1 to 12, wherein: The camera includes an integrated chip on which the circuit is arranged, and the integrated chip further includes a memory storing a predetermined algorithm for image processing of the circuit.

Images