Camera monitoring method and device, storage medium and computer program product

Abstract

The invention discloses a camera monitoring method and device, a storage medium and a computer program product. The method comprises the following steps: in response to a video acquisition request, acquiring a video image stream acquired by a camera, and performing feature extraction on image frames in the video image stream to obtain a first feature map; target bounding boxes are determined in the image frame according to the first feature map, and each target bounding box corresponds to one target object; in response to the target selection instruction, determining a target tracking object planned to be tracked in the plurality of target objects, and determining state information of the target tracking object in the image frame; and according to the state information of the target tracking object, determining the motion trail of the target object, and according to the motion trail, sending a control instruction to the camera, the control instruction being used for indicating the camera to adjust the shooting angle and direction. According to the invention, the technical problem that the intelligent degree and the automation degree of a visual intercom pan-tilt camera in the related technology are insufficient is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a camera monitoring method, device, storage medium, and computer program product. Background Art

[0002] With the rapid development of smart homes and smart security systems, video intercom cameras have become an indispensable part of modern home and commercial environments. Traditional video intercom systems mainly provide basic video calls and monitoring functions, but with the advancement of technology, users have higher and higher requirements for smart devices, expecting them to provide more personalized and intelligent services. However, although the video intercom camera system in the relevant technology has basic motion detection and video call functions, it still has a lot of room for improvement in terms of intelligence and automation.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a camera monitoring method, device, storage medium and computer program product to at least solve the technical problem of insufficient intelligence and automation of video intercom pan-tilt cameras in the related art.

[0005] According to one aspect of an embodiment of the present application, a camera monitoring method is provided, including: in response to a video acquisition request, obtaining a video image stream captured by a camera, and performing feature extraction on image frames in the video image stream to obtain a first feature map; based on the first feature map, determining a target bounding box in the image frame, wherein the target bounding box is used to indicate an area in the image frame where a target object is located, and each target bounding box corresponds to a target object; in response to a target selection instruction, determining a target tracking object to be tracked among multiple target objects, and determining state information of the target tracking object in the image frame, wherein the state information includes: speed and position; based on the state information of the target tracking object, determining a motion trajectory of the target object, and based on the motion trajectory, sending a control instruction to the camera, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

[0006] Optionally, before extracting features from image frames in a video image stream, the method further includes: converting the image frames into grayscale images, and counting the number of pixels at different brightness levels in the grayscale images to obtain a first brightness distribution histogram; determining the total number of pixels in the image frame, and normalizing the data of the first brightness distribution histogram based on the total number to obtain a second brightness distribution histogram; determining the brightness state of the image frame based on the second brightness distribution histogram, wherein the brightness state includes: dark, bright, and balanced; adjusting based on the brightness state, including: when the brightness state is bright, reducing the aperture of the camera and / or increasing the shutter speed of the camera, and, when the brightness state is dark, increasing the aperture of the camera and / or slowing down the shutter speed of the camera.

[0007] Optionally, the method also includes: converting the image frame to a target color space, wherein the target color space includes: a YCbCr color space; determining a chromaticity component value corresponding to the image frame in the target color space, and determining a gain factor corresponding to each color channel based on the chromaticity component, wherein the color channel includes: a red channel, a green channel, and a blue channel; adjusting the image frame based on the gain factors corresponding to different color channels, wherein the gain factors are used to adjust the color temperature and hue of the image frame.

[0008] Optionally, performing feature extraction on image frames in a video image stream to obtain a first feature map includes: performing feature extraction on the image frames to obtain a second feature map, and using a feature pyramid network to fuse second feature maps of different scales to obtain a first feature map; determining a target bounding box in the image frame based on the first feature map includes: determining a position offset of an anchor box based on the first feature map, and adjusting a position of the anchor box based on the position offset to obtain a target bounding box, wherein the anchor box is a pre-set border in the image frame.

[0009] Optionally, determining the motion trajectory of the target object based on the state information of the target tracking object includes: using a Kalman filter to predict a predicted bounding box of the target tracking object in a second image frame based on the state information of the target bounding box of the target tracking object in the first image frame, wherein the second image frame is an image frame immediately following the first image frame in the video image stream; obtaining the target bounding box detected in the second image frame, and constructing a cost matrix between each target bounding box in the second image frame and the predicted bounding box, wherein each element in the cost matrix is ​​used to characterize the matching cost between the predicted bounding box and each target bounding box detected in the second image frame, and the matching cost includes: feature similarity, distance deviation, and border shape similarity; based on the cost matrix, determining a target bounding box that matches the predicted bounding box among each target bounding box detected in the second image frame, and updating the predicted bounding box based on the target bounding box to obtain a target tracking box corresponding to the target tracking object in the second image frame, and updating the Kalman gain value in the Kalman filter based on the target bounding box.

[0010] Optionally, the method also includes: determining the facial feature points of the target object in the image frame in the video image stream based on the first feature map, and determining the geometric features corresponding to each image frame based on the facial feature points, wherein the geometric features are used to characterize the position distance relationship between the eyes, nose, and mouth of the target object; using an inter-frame difference method to determine the expression amplitude change characteristics of the target object in the video image stream based on the geometric features corresponding to each image frame; extracting the voice signal corresponding to the video image stream, and determining the tone change characteristics of the target object based on the voice signal; using a support vector machine to determine the emotional state of the target object based on the expression amplitude change characteristics and the tone change characteristics.

[0011] Optionally, determining the tone change characteristics of the target object based on the speech signal includes: dividing the speech signal into short frames of a preset length, and performing Fourier transform on each short frame to obtain corresponding spectrum information; converting the spectrum information into Mel frequency data, and extracting the tone change characteristics in the Mel frequency data, wherein the tone change characteristics include: first-order difference and second-order difference characteristics of the Mel frequency data for characterizing the changing trend of the speech signal, fundamental frequency characteristics for characterizing the changing range and changing speed of the pitch, syllable rate characteristics for characterizing the speed of speech, and rhythm characteristics for characterizing the rhythm characteristics of the speech.

[0012] According to another aspect of an embodiment of the present application, a camera monitoring device is also provided, including: a video stream acquisition module, used to respond to a video acquisition request, acquire a video image stream captured by a camera, and perform feature extraction on image frames in the video image stream to obtain a first feature map; a target detection module, used to determine a target bounding box in the image frame based on the first feature map, wherein the target bounding box is used to indicate an area in the image frame where a target object is located, and each target bounding box corresponds to a target object; an object selection module, used to determine a target tracking object to be tracked among multiple target objects in response to a target selection instruction, and determine state information of the target tracking object in the image frame, wherein the state information includes: speed and position; a target tracking module, used to determine a motion trajectory of the target object based on the state information of the target tracking object, and send a control instruction to the camera based on the motion trajectory, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

[0013] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the camera monitoring method by running the computer program.

[0014] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the camera monitoring method are implemented.

[0015] In an embodiment of the present application, in response to a video acquisition request, a video image stream captured by a camera is acquired, and feature extraction is performed on image frames in the video image stream to obtain a first feature map; based on the first feature map, a target bounding box is determined in the image frame, wherein the target bounding box is used to indicate the area where the target object is located in the image frame, and each target bounding box corresponds to a target object; in response to a target selection instruction, a target tracking object to be tracked is determined from multiple target objects, and state information of the target tracking object in the image frame is determined, wherein the state information includes: speed and position; based on the state information of the target tracking object, a motion trajectory of the target object is determined, and based on the motion trajectory, a control instruction is sent to the camera, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction. Through the intelligent video intercom camera system with integrated environmental adaptability adjustment function, the camera angle can be automatically adjusted to track the target object in the video and analyze it, thereby achieving the purpose of providing a more personalized and intelligent user experience, thereby solving the technical problem of insufficient intelligence and automation of video intercom pan-tilt cameras in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 It is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a camera monitoring method provided in an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a camera monitoring method flow according to an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of a method flow of a computer vision-based visual intercom PTZ camera monitoring method provided in accordance with an embodiment of the present application;

[0020] Figure 4 It is a structural schematic diagram of a camera monitoring device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] With the rapid development of smart homes and smart security systems, video intercom cameras have become an indispensable part of modern homes and commercial environments. For example, smart security cameras can identify abnormal behavior and sound alarms to improve safety; smart driving cameras can detect the driver's status and prevent accidents. The video intercom PTZ camera combines camera functions and real-time intercom technology. It not only provides high-definition video monitoring, but also enables remote intercom through the built-in two-way audio system. Users can easily view real-time images through mobile apps or computers and communicate instantly with people at the camera location. In the home, the video intercom camera can achieve remote care, home security monitoring and communication with family members; in commercial places, it is often used to monitor the safety of employees, customers or materials to improve management efficiency and safety.

[0024] In summary, the traditional video intercom system in the relevant technology mainly provides basic video call and monitoring functions, but with the advancement of technology, users have higher and higher requirements for smart devices, expecting them to provide more personalized and intelligent services. Although the video intercom camera system on the market currently has basic motion detection and video call functions, there is still a lot of room for improvement in voice control, emotion recognition and environmental adaptability.

[0025] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0026] According to an embodiment of the present application, a method embodiment of camera monitoring is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a camera monitoring method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0028] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the camera monitoring method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned camera monitoring method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0032] In the above operating environment, the embodiment of the present application provides a camera monitoring method. Figure 2 is a schematic diagram of a camera monitoring method flow according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0033] Step S202, in response to the video acquisition request, acquiring a video image stream captured by a camera, and performing feature extraction on image frames in the video image stream to obtain a first feature map;

[0034] Step S204, determining a target bounding box in the image frame according to the first feature map, wherein the target bounding box is used to indicate an area where the target object is located in the image frame, and each target bounding box corresponds to a target object;

[0035] Step S206, in response to the target selection instruction, determining a target tracking object to be tracked from among the multiple target objects, and determining state information of the target tracking object in the image frame, wherein the state information includes: speed and position;

[0036] Step S208, determining the motion trajectory of the target object according to the state information of the target tracking object, and sending a control instruction to the camera according to the motion trajectory, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

[0037] Through the above steps, the intelligent video intercom camera system with integrated environmental adaptability adjustment function can automatically adjust the camera angle to track the target object in the video and perform analysis, thereby achieving the purpose of providing a more personalized and intelligent user experience, thereby solving the technical problem of insufficient intelligence and automation of video intercom pan-tilt cameras in related technologies.

[0038] The following further introduces the camera monitoring method in steps S202 to S208 of the embodiment of the present application.

[0039] Figure 3 is a schematic diagram of a method flow of a computer vision-based visual intercom PTZ camera monitoring method provided in an embodiment of the present application, such as Figure 3 As shown, the embodiment of the present application provides an intelligent video intercom camera monitoring method and system that integrates emotion recognition and environmental adaptability adjustment functions, which can automatically adjust the camera angle to track the target object in the video and perform emotion analysis on the target. Through these functions, a more personalized and intelligent user experience can be provided, thereby realizing a more intelligent and automated monitoring solution, which is described in detail below.

[0040] When a video acquisition request is received, the video image stream captured by the camera is obtained. Specifically, the user can call the video intercom camera through a mobile phone application or a small program. After the video intercom camera is connected through a physical button or a screen button, a video call can be realized, and the video image stream captured by the camera is obtained. In an embodiment of the present application, the camera can make adaptive adjustments according to the current image environment when capturing video images, as follows.

[0041] In this embodiment, the brightness distribution can be determined by calculating the brightness histogram of the video frame; the aperture size and exposure time are adjusted according to the brightness distribution to achieve automatic exposure adjustment. The specific steps are as follows.

[0042] In some embodiments of the present application, before extracting features from image frames in a video image stream, the method also includes the following steps: converting the image frame into a grayscale image, and counting the number of pixels at different brightness levels in the grayscale image to obtain a first brightness distribution histogram; determining the total number of pixels in the image frame, and normalizing the data of the first brightness distribution histogram based on the total number to obtain a second brightness distribution histogram; determining the brightness state of the image frame based on the second brightness distribution histogram, wherein the brightness state includes: dark, bright, and balanced; adjusting based on the brightness state, including: when the brightness state is bright, reducing the aperture of the camera and / or increasing the shutter speed of the camera, and, when the brightness state is dark, increasing the aperture of the camera and / or slowing down the shutter speed of the camera.

[0043] Specifically, first, a separate image frame is extracted from the video image stream, usually represented in the form of a matrix. Since the histogram is usually based on brightness information, the image frame in RGB color image format needs to be converted into a grayscale image, as shown in the following formula:

[0044] I gray (x,y)=0.299R(x,y)+0.587G(x,y)+0.114B(x,y)

[0045] Among them, I gray (x, y) is the pixel value in the grayscale image, and R(x, y), G(x, y), and B(x, y) are the red, green, and blue components in the RGB image, respectively.

[0046] After that, the brightness level is determined according to the number of bits in the image. For example, an 8-bit image has 256 brightness levels (0 to 255). For each pixel in the grayscale image, the count at the corresponding brightness level is increased, as shown in the following formula:

[0047]

[0048] Wherein, H(i) is the pixel count of brightness level i, that is, the number of pixels at brightness level i in the grayscale image, W and H are the width and height of the image respectively, δ is the Kronecker function, which is 1 when the condition is met, otherwise it is 0. After the number of pixels at different brightness levels is counted, the above-mentioned first brightness distribution histogram can be obtained.

[0049] Then, the histogram is normalized using the total number of pixels in the image frame to obtain a second brightness distribution histogram. Specifically, each value of the histogram is divided by the total number of pixels in the image to obtain a probability distribution: Among them, H′(i) is the normalized histogram value.

[0050] By analyzing the second brightness distribution histogram, the brightness state of the image frame can be determined and further exposure adjustment can be performed. Specifically, the peak on the left side of the histogram indicates that the image is dark, the peak on the right side indicates that the image is bright, and the uniform distribution of the histogram indicates that the image brightness is balanced.

[0051] If the image is too bright, you can increase the f-stop value to reduce the aperture, reduce the amount of light entering, and / or use a faster shutter speed to reduce the exposure time; if the image is too dark, you can reduce the f-stop value to increase the aperture, increase the amount of light entering, and / or use a slower shutter speed to increase the exposure time. Among them, f-stop = lens focal length / aperture diameter, shutter speed = 1 / frame rate.

[0052] Additionally, you can use exposure compensation to fine-tune the brightness of the image: EV 补偿 =EV 原始 + Compensation value, where EV 原始 is the original exposure value, and the compensation value is a positive or negative number set by the user. After applying the adjustment, you can capture new image samples to verify the effect of the adjustment. If the exposure of the new image is still not ideal, continue the iterative adjustment process.

[0053] In addition, you can also calculate the color temperature and hue of the image, automatically adjust the gain of the RGB channels, and adjust the white balance. The specific steps are as follows.

[0054] In some embodiments of the present application, the method also includes the following steps: converting the image frame to a target color space, wherein the target color space includes: a YCbCr color space; determining the chromaticity component value corresponding to the image frame in the target color space, and determining the gain factor corresponding to each color channel based on the chromaticity component, wherein the color channel includes: a red channel, a green channel, and a blue channel; adjusting the image frame based on the gain factors corresponding to different color channels, wherein the gain factors are used to adjust the color temperature and hue of the image frame.

[0055] Automatic white balance adjustment is an important technology in image processing. Its purpose is to ensure that white objects in the image appear as true white, so that the color of the entire image looks more natural. The specific steps are as follows.

[0056] First, the image frame is converted from the RGB color space to a target color space more suitable for white balance adjustment, such as YCbCr or XYZ. In the YCbCr color space, chrominance is represented by Cb and Cr components as shown below:

[0057] Y=0.299R+0.587G+0.114B

[0058]

[0059] Where B, G, R are the values ​​of the blue, green, and red channels of the original image, respectively, and Y is the brightness component. Set the target chromaticity to 128 (in the range of 0-255), representing a neutral chromaticity: Cb target =128,Cr target =128, then the gain factor of each color channel can be calculated according to the difference between the chromaticity of the image and the target chromaticity, as shown in the following formula:

[0060]

[0061] Apply the gain factor to each color channel to adjust the color temperature and hue of the image. Then convert the adjusted image from the YCbCr or XYZ color space back to the RGB color space as shown in the following formula:

[0062] R adjusted =R×R gain

[0063] G adjusted =G×G gain

[0064] B adjusted =B×B gain

[0065] This application solution integrates automatic exposure and white balance adjustment functions, analyzes brightness distribution by calculating the brightness histogram of the video frame, intelligently adjusts the aperture size and exposure time, and automatically adjusts the gain of the RGB channel according to the color temperature and hue of the image. It significantly improves the quality of video images under different lighting conditions, ensures the true restoration of colors and image clarity, and brings users a more stable and natural visual experience.

[0066] After adaptively adjusting the image frames in the video image stream, target object detection can be performed on them. In this embodiment, YOLOv4 or SSD model can be used for real-time object detection. The model structure includes: input layer, multiple convolutional layers, feature pyramid network and detection layer. The specific process steps are as follows.

[0067] In some embodiments of the present application, performing feature extraction on image frames in a video image stream to obtain a first feature map includes the following steps: performing feature extraction on the image frame to obtain a second feature map, and using a feature pyramid network to fuse the second feature maps of different scales to obtain a first feature map; based on the first feature map, determining a target bounding box in the image frame includes the following steps: determining a position offset of an anchor box based on the first feature map, and adjusting the position of the anchor box based on the position offset to obtain a target bounding box, wherein the anchor box is a pre-set border in the image frame.

[0068] Specifically, in the input layer, the model receives an input image, which usually needs to be resized to the fixed size used when the model was trained;

[0069] In the convolution layer, the local features of the image are extracted through the convolution operation of multiple convolution layers (possibly including depth-separable convolution) to form a second feature map. Applying batch normalization after the convolution layer helps to speed up the training process and reduce the model's sensitivity to initialization weights. Usually, activation functions such as ReLU or Mish can be used to introduce nonlinearity to help the model learn more complex features;

[0070] Afterwards, the feature pyramid network captures multi-scale features of different scales through the FPN structure by combining the second feature maps of the low layer (high resolution but shallow layer) and the high layer (low resolution but deep layer). Starting from the high-level features of the basic network, it is passed down layer by layer from top to bottom, and the high-level features are combined with the low-level features through skip connections. Lateral connections are applied to each feature map to enhance the representation ability of the features and obtain the first feature map. This structure enhances the model's detection ability for objects of different sizes, making it more suitable for subsequent detection tasks.

[0071] Finally, the detection layer predicts the position offset (predicts the center coordinate offset of each anchor box), confidence (predicts the probability that each anchor box contains an object and the probability that the object belongs to the background) and category probability (predicts the category probability of the object in each anchor box) of the anchor box on each first feature map. Bounding box regression: Use the predicted position offset to adjust the position of the anchor box to generate the final bounding box. Apply NMS (Non-Maximum Suppression) to the detected bounding boxes, remove overlapping bounding boxes, retain the best prediction results, and filter out predictions with confidence levels below the threshold to reduce false detections. Finally, draw the final target bounding box on the original image, and annotate the category and confidence of the detected object to achieve fast and accurate detection of multiple target objects in the image.

[0072] In the process of target object detection, the present application scheme uses a target detection method driven by a deep learning model and combines a feature pyramid network to extract multi-scale features, thus achieving fast and accurate detection of target objects of different sizes in the image. It can effectively identify and locate target objects in images, regardless of their size, and maintain high accuracy, significantly improving the response speed and detection quality of the monitoring system.

[0073] After determining the target detection frames corresponding to multiple target objects in the image frame, the target object to be tracked in the image can be selected according to the confidence threshold and user interaction, that is, in response to the target selection instruction, a target tracking object planned to be tracked is determined from multiple target objects. After that, the camera can continuously monitor the changes in the field of view during the video intercom video call, detect whether the target object to be tracked moves through pixel-level comparison, identify the state information of the moving target (including but not limited to: position and speed), and thus start the tracking function. The system predicts its possible motion trajectory and adjusts the camera's viewing angle and focus in time to ensure that the target is always in the field of view. The specific steps are as follows.

[0074] In some embodiments of the present application, determining the motion trajectory of the target object based on the state information of the target tracking object includes the following steps: using a Kalman filter to predict the predicted bounding box of the target tracking object in the second image frame based on the state information of the target bounding box of the target tracking object in the first image frame, wherein the second image frame is an image frame immediately following the first image frame in the video image stream; obtaining the target bounding box detected in the second image frame, and constructing a cost matrix between each target bounding box in the second image frame and the predicted bounding box, wherein each element in the cost matrix is ​​used to characterize the matching cost between the predicted bounding box and each target bounding box detected in the second image frame, and the matching cost includes: feature similarity, distance deviation, and border shape similarity; based on the cost matrix, determining the target bounding box that matches the predicted bounding box among each target bounding box detected in the second image frame, and updating the predicted bounding box based on the target bounding box to obtain the target tracking box corresponding to the target tracking object in the second image frame, and updating the Kalman gain value in the Kalman filter based on the target bounding box.

[0075] Specifically, a feature extractor is used to extract features of the target within the target tracking box of the target object to be tracked using a deep learning model, which may be deep features extracted by CNN, such as feature maps output by a specific layer of ResNet or DenseNet, and these features are used in the subsequent matching process.

[0076] The Kalman filter is used to predict the state (position, speed, etc.) of each target. In the absence of observation data, its future state is predicted based on the target's motion model, as shown in the following formula:

[0077]

[0078] When new observation data arrives, the Kalman filter updates the estimate of the target state based on the observation, as shown in the following formula:

[0079]

[0080] in, is the state estimate, Information representing the state estimate of the target bounding box in the first image frame, The predicted state of the predicted bounding box of the corresponding predicted target tracking object in the second image frame, The state of the target tracking box corresponding to the target tracking object in the second image frame, A is the state transfer matrix, B is the control matrix, ut is the control input, wt is the process noise, Kt is the Kalman gain, zt is the observation value, and H is the observation matrix.

[0081] In an embodiment of the present application, the Hungarian algorithm can be used to solve the data association problem, that is, how to match the detection result of the current frame (the second image frame) with the tracking target of the previous frame (the first image frame). Specifically, a cost matrix (cost matrix) can be constructed, in which the elements represent the cost of matching, such as based on feature distance, position distance or confidence, and then the Hungarian algorithm is used to find the matching solution that minimizes the total cost in the cost matrix (cost matrix), and the tracking status of each target is updated according to the matching results.

[0082] For example, a unique tracking ID can be assigned to each newly detected target. The tracking state of each target is maintained, including position, velocity, appearance features, and tracking ID. The target loss and re-identification problems are also handled, such as when the target temporarily leaves the field of view or is occluded. Finally, the tracking results, including the bounding box and tracking ID, are drawn on the video frame or image, and the position, tracking ID, and other related information of each tracked target are provided.

[0083] Through the above steps, multiple targets in the video can be effectively tracked, and accurate matching and status updates can be performed between targets. After that, the PID controller can be used to adjust the angle of the PTZ camera according to the target's motion trajectory, that is, according to the motion trajectory, a control instruction is sent to the camera, where the control instruction is used to instruct the camera to adjust the shooting angle and direction, as shown in the following formula:

[0084]

[0085] Among them, θ t+1 is the camera angle at the next moment, Kp, Ki, and Kd are all parameters of the PID controller, and et is the error, that is, the deviation between the position of the next frame in the motion trajectory of the target tracking object and the center point of the picture field captured by the camera in the current frame.

[0086] In the target tracking step, the present application scheme uses a feature extractor to extract the features of the target object, and predicts the target state with the help of a Kalman filter, while using the Hungarian algorithm for data association and matching to achieve stable tracking of the target object. Through precise feature matching and state prediction, it is ensured that the target can be accurately tracked even when the target moves or the environment changes, thereby improving the reliability and practicality of the system.

[0087] In order to further improve the intelligence level of the video intercom PTZ camera monitoring process, the embodiment of the present application can also perform facial expression recognition and voice emotion analysis on the tracked object during the target tracking process to analyze the emotional state of the target object during the video call. The specific steps are as follows.

[0088] In some embodiments of the present application, the method also includes the following steps: determining the facial feature points of the target object in the image frame in the video image stream based on the first feature map, and determining the geometric features corresponding to each image frame based on the facial feature points, wherein the geometric features are used to characterize the position distance relationship between the eyes, nose, and mouth of the target object; using the inter-frame difference method, based on the geometric features corresponding to each image frame, determining the expression amplitude change characteristics of the target object in the video image stream; extracting the voice signal corresponding to the video image stream, and determining the tone change characteristics of the target object based on the voice signal; using a support vector machine, based on the expression amplitude change characteristics and the tone change characteristics, determining the emotional state of the target object.

[0089] In this embodiment, facial key points and expression features can be extracted based on the first feature map. Specifically, the video is decomposed into an image frame sequence, each frame of the video image is preprocessed in chronological order of the video clips, the distance between the centers of the pupils of both eyes of the face is calculated, and the distance between the centers of the pupils of both eyes is uniformly scaled to a distance of a preset pixel size; the image is cropped into an M×M pixel area image containing key expression parts such as the mouth, nose, and forehead, to obtain an image frame sequence with normalized face image size.

[0090] Then, the facial feature points of the image frame sequence are marked, and the key frames in the image frame sequence are selected. The feature point detection algorithm, such as dlib's 68-point model or a deep learning-based model, is applied to the aligned face images to mark the key feature points of the face. Features are extracted from the marked feature points, such as geometric features (the positional relationship of the eyes, nose, and mouth) and texture features (pixel changes around the feature points); the difference between consecutive frames is calculated using the frame difference method or the optical flow method, and the motion amplitude of the face in each frame is evaluated, such as calculating the change in the position of the feature points. According to the results of the frame difference and motion evaluation, a threshold is set to determine the key frame. Among them, the key frame can be a frame with significant motion changes or a frame with significant facial feature changes. These frames can represent important moments or expression changes in the video, that is, expression amplitude change features.

[0091] On the other hand, the emotional features of speech signals can also be extracted. For example, the speech signal in the video image stream can be extracted and saved in MP3 format, speech detection can be performed, and noise and silence segments can be removed to obtain a speech signal that is easier to extract features. The specific steps are as follows.

[0092] In some embodiments of the present application, determining the tone change characteristics of the target object based on the speech signal includes the following steps: dividing the speech signal into short frames of a preset length, and performing Fourier transform on each short frame to obtain corresponding spectral information; converting the spectral information into Mel frequency data, and extracting the tone change characteristics in the Mel frequency data, wherein the tone change characteristics include: first-order difference and second-order difference characteristics of the Mel frequency data for characterizing the changing trend of the speech signal, fundamental frequency characteristics for characterizing the changing range and speed of the pitch, syllable rate characteristics for characterizing the speed of speech, and rhythm characteristics for characterizing the rhythm characteristics of speech.

[0093] Specifically, the continuous speech signal is divided into short frames (usually 20-40 milliseconds) to adapt to the non-stationary characteristics of speech, and each frame is subjected to a fast Fourier transform (FFT) to obtain spectral information. The output of the Mel filter bank is calculated, and the spectrum obtained by FFT is converted to the Mel frequency scale (i.e., Mel frequency data): MFCC = log (DCT (FB (s))), where S is the original speech signal (spectral information), FB is the Mel filter bank, and DCT is the discrete cosine transform.

[0094] Afterwards, the dynamic change characteristics of the speech signal are extracted, including but not limited to: the first-order and second-order differences of MFCC, which are used to reflect the change trend of the speech signal; the YIN algorithm is used to detect the fundamental frequency (pitch) of the speech signal and extract the change characteristics of the pitch, such as the change range and change speed of the pitch; the rhythm characteristics are determined by calculating the rate of syllables in the speech (reflecting the speed of speaking) and extracting the duration and frequency of pauses (reflecting the rhythm characteristics of the speech).

[0095] Finally, the support vector machine (SVM) is used to classify the emotional state of the tracked object during the video call based on the expression amplitude change characteristics of the key frames and the tone change characteristics of the voice data. Specifically, the SVM-based system can classify the emotional state of the tracked object during the video call based on the key frame characteristics and the dynamic change characteristics of the voice data, which can be applied to automatic customer service, sentiment analysis research, health monitoring and other fields.

[0096] This application scheme combines facial key point extraction and speech emotion feature extraction technology, identifies emotional states by analyzing facial expressions and speech signals in videos, and uses support vector machines (SVM) for classification, thereby realizing intelligent analysis of the emotional states of target objects. It can provide users with more personalized and considerate services by gaining insight into the emotional changes of users, and also provides valuable data support for research and application in related fields.

[0097] According to an embodiment of the present application, an embodiment of a camera monitoring device is also provided. Figure 4Schematic diagram of the structure of a camera monitoring device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0098] The video stream acquisition module 40 is used to obtain the video image stream captured by the camera in response to the video acquisition request, and perform feature extraction on the image frames in the video image stream to obtain a first feature map;

[0099] A target detection module 42, configured to determine a target bounding box in the image frame according to the first feature map, wherein the target bounding box is used to indicate an area in the image frame where a target object is located, and each target bounding box corresponds to a target object;

[0100] The object selection module 44 is used to determine a target tracking object to be tracked among multiple target objects in response to the target selection instruction, and determine the state information of the target tracking object in the image frame, wherein the state information includes: speed and position;

[0101] The target tracking module 46 is used to determine the motion trajectory of the target object according to the state information of the target tracking object, and send a control instruction to the camera according to the motion trajectory, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

[0102] Optionally, before extracting features from image frames in the video image stream, the video stream acquisition module 40 is also used to: convert the image frames into grayscale images, and count the number of pixels at different brightness levels in the grayscale images to obtain a first brightness distribution histogram; determine the total number of pixels in the image frame, and normalize the data of the first brightness distribution histogram based on the total number to obtain a second brightness distribution histogram; determine the brightness state of the image frame based on the second brightness distribution histogram, wherein the brightness state includes: dark, bright, and balanced; adjust according to the brightness state, including: when the brightness state is bright, reducing the aperture of the camera and / or increasing the shutter speed of the camera, and, when the brightness state is dark, increasing the aperture of the camera and / or slowing down the shutter speed of the camera.

[0103] Optionally, the video stream acquisition module 40 is also used to: convert the image frame to a target color space, wherein the target color space includes: a YCbCr color space; determine the chromaticity component value corresponding to the image frame in the target color space, and determine the gain factor corresponding to each color channel based on the chromaticity component, wherein the color channel includes: a red channel, a green channel, and a blue channel; adjust the image frame based on the gain factors corresponding to different color channels, wherein the gain factors are used to adjust the color temperature and hue of the image frame.

[0104] Optionally, performing feature extraction on image frames in a video image stream to obtain a first feature map includes: performing feature extraction on the image frames to obtain a second feature map, and using a feature pyramid network to fuse second feature maps of different scales to obtain a first feature map; determining a target bounding box in the image frame based on the first feature map includes: determining a position offset of an anchor box based on the first feature map, and adjusting a position of the anchor box based on the position offset to obtain a target bounding box, wherein the anchor box is a pre-set border in the image frame.

[0105] Optionally, determining the motion trajectory of the target object based on the state information of the target tracking object includes: using a Kalman filter to predict a predicted bounding box of the target tracking object in a second image frame based on the state information of the target bounding box of the target tracking object in the first image frame, wherein the second image frame is an image frame immediately following the first image frame in the video image stream; obtaining the target bounding box detected in the second image frame, and constructing a cost matrix between each target bounding box in the second image frame and the predicted bounding box, wherein each element in the cost matrix is ​​used to characterize the matching cost between the predicted bounding box and each target bounding box detected in the second image frame, and the matching cost includes: feature similarity, distance deviation, and border shape similarity; based on the cost matrix, determining a target bounding box that matches the predicted bounding box among each target bounding box detected in the second image frame, and updating the predicted bounding box based on the target bounding box to obtain a target tracking box corresponding to the target tracking object in the second image frame, and updating the Kalman gain value in the Kalman filter based on the target bounding box.

[0106] Optionally, the camera monitoring device is also used to: determine the facial feature points of the target object in the image frame in the video image stream based on the first feature map, and determine the geometric features corresponding to each image frame based on the facial feature points, wherein the geometric features are used to characterize the position distance relationship between the eyes, nose and mouth of the target object; use the inter-frame difference method to determine the expression amplitude change characteristics of the target object in the video image stream based on the geometric features corresponding to each image frame; extract the voice signal corresponding to the video image stream, and determine the tone change characteristics of the target object based on the voice signal; use a support vector machine to determine the emotional state of the target object based on the expression amplitude change characteristics and the tone change characteristics.

[0107] Optionally, determining the tone change characteristics of the target object based on the speech signal includes: dividing the speech signal into short frames of a preset length, and performing Fourier transform on each short frame to obtain corresponding spectrum information; converting the spectrum information into Mel frequency data, and extracting the tone change characteristics in the Mel frequency data, wherein the tone change characteristics include: first-order difference and second-order difference characteristics of the Mel frequency data for characterizing the changing trend of the speech signal, fundamental frequency characteristics for characterizing the changing range and changing speed of the pitch, syllable rate characteristics for characterizing the speed of speech, and rhythm characteristics for characterizing the rhythm characteristics of the speech.

[0108] It should be noted that the various modules in the above-mentioned camera monitoring device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0109] It should be noted that the camera monitoring device provided in this embodiment can be used to perform Figure 2 The camera monitoring method shown, therefore, the relevant explanations and descriptions of the above-mentioned camera monitoring method are also applicable to the embodiments of the present application and will not be repeated here.

[0110] An embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following camera monitoring method by running the computer program: in response to a video acquisition request, obtain a video image stream captured by a camera, and perform feature extraction on image frames in the video image stream to obtain a first feature map; based on the first feature map, determine a target bounding box in the image frame, wherein the target bounding box is used to indicate an area where a target object is located in the image frame, and each target bounding box corresponds to a target object; in response to a target selection instruction, determine a target tracking object to be tracked among multiple target objects, and determine the state information of the target tracking object in the image frame, wherein the state information includes: speed and position; based on the state information of the target tracking object, determine the motion trajectory of the target object, and based on the motion trajectory, send a control instruction to the camera, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

[0111] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the camera monitoring method described in each embodiment of the present application: in response to a video acquisition request, obtaining a video image stream captured by a camera, and performing feature extraction on image frames in the video image stream to obtain a first feature map; based on the first feature map, determining a target bounding box in the image frame, wherein the target bounding box is used to indicate an area in the image frame where a target object is located, and each target bounding box corresponds to a target object; in response to a target selection instruction, determining a target tracking object to be tracked among multiple target objects, and determining state information of the target tracking object in the image frame, wherein the state information includes: speed and position; based on the state information of the target tracking object, determining a motion trajectory of the target object, and based on the motion trajectory, sending a control instruction to the camera, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

[0112] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0113] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0116] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0118] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A camera monitoring method, characterized in that: include: In response to a video acquisition request, a video image stream captured by a camera is acquired, and feature extraction is performed on image frames in the video image stream to obtain a first feature map; Determine a target bounding box in the image frame according to the first feature map, wherein the target bounding box is used to indicate an area in the image frame where a target object is located, and each target bounding box corresponds to a target object; In response to the target selection instruction, a target tracking object to be tracked is determined among the multiple target objects, and state information of the target tracking object in the image frame is determined, wherein the state information includes: speed and position; The motion trajectory of the target object is determined according to the state information of the target tracking object, and a control instruction is sent to the camera according to the motion trajectory, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

2. The camera monitoring method according to claim 1, characterized in that: Before extracting features from the image frames in the video image stream, the method further includes: Converting the image frame into a grayscale image, and counting the number of pixels at different brightness levels in the grayscale image to obtain a first brightness distribution histogram; Determining the total number of pixels in the image frame, and normalizing the data of the first brightness distribution histogram according to the total number to obtain a second brightness distribution histogram; Determining the brightness state of the image frame according to the second brightness distribution histogram, wherein the brightness state includes: dark, bright, and balanced; Adjustment is performed according to the brightness state, including: when the brightness state is bright, reducing the aperture of the camera and / or increasing the shutter speed of the camera; and when the brightness state is dark, increasing the aperture of the camera and / or slowing down the shutter speed of the camera.

3. The camera monitoring method according to claim 2, characterized in that: The method further comprises: Converting the image frame to a target color space, wherein the target color space includes: a YCbCr color space; Determine the chrominance component value corresponding to the image frame in the target color space, and determine the gain factor corresponding to each color channel according to the chrominance component, wherein the color channels include: a red channel, a green channel, and a blue channel; The image frame is adjusted according to the gain factors corresponding to different color channels, wherein the gain factors are used to adjust the color temperature and hue of the image frame.

4. The camera monitoring method according to claim 1, characterized in that: Extracting features from the image frames in the video image stream to obtain the first feature map includes: extracting features from the image frames to obtain a second feature map, and fusing the second feature maps of different scales using a feature pyramid network to obtain the first feature map; Determining a target bounding box in the image frame based on the first feature map includes: determining a position offset of an anchor box based on the first feature map, and adjusting a position of the anchor box based on the position offset to obtain the target bounding box, wherein the anchor box is a pre-set border in the image frame.

5. The camera monitoring method according to claim 1, characterized in that: Determining the motion trajectory of the target object according to the state information of the target tracking object includes: Using a Kalman filter, based on the state information of the target bounding box of the target tracking object in the first image frame, predicting the predicted bounding box of the target tracking object in the second image frame, wherein the second image frame is an image frame immediately following the first image frame in the video image stream; Acquire the target bounding box detected in the second image frame, and construct a cost matrix between each target bounding box in the second image frame and the predicted bounding box, wherein each element in the cost matrix is ​​used to represent the matching cost between the predicted bounding box and each target bounding box detected in the second image frame, and the matching cost includes: feature similarity, distance deviation, and border shape similarity; According to the cost matrix, a target bounding box matching the predicted bounding box is determined in each target bounding box detected in the second image frame, and the predicted bounding box is updated according to the target bounding box to obtain the target tracking box corresponding to the target tracking object in the second image frame, and the Kalman gain value in the Kalman filter is updated according to the target bounding box.

6. The camera monitoring method according to claim 1, characterized in that: The method further comprises: Determine facial feature points of the target object in the image frames in the video image stream based on the first feature map, and determine geometric features corresponding to each of the image frames based on the facial feature points, wherein the geometric features are used to characterize the positional distance relationship between the eyes, nose, and mouth of the target object; Using an inter-frame difference method, according to the geometric features corresponding to each of the image frames, determine the expression amplitude variation features of the target object in the video image stream; Extracting a voice signal corresponding to the video image stream, and determining the tone change characteristics of the target object based on the voice signal; A support vector machine is used to determine the emotional state of the target object according to the expression amplitude change characteristics and the tone change characteristics.

7. The camera monitoring method according to claim 6, characterized in that: Determining the tone change feature of the target object according to the speech signal includes: Dividing the speech signal into short frames of a preset length, and performing Fourier transform on each of the short frames to obtain corresponding spectrum information; The spectral information is converted into Mel frequency data, and the tone change features in the Mel frequency data are extracted, wherein the tone change features include: first-order difference and second-order difference features of the Mel frequency data for characterizing the change trend of the speech signal, fundamental frequency features for characterizing the change range and change speed of the pitch, syllable rate features for characterizing the speed of speech, and rhythm features for characterizing the rhythm characteristics of the speech.

8. A camera monitoring device, characterized in that: include: A video stream acquisition module, used to obtain a video image stream captured by a camera in response to a video acquisition request, and perform feature extraction on image frames in the video image stream to obtain a first feature map; a target detection module, configured to determine a target bounding box in the image frame according to the first feature map, wherein the target bounding box is used to indicate an area in the image frame where a target object is located, and each target bounding box corresponds to a target object; The object selection module is used to determine a target tracking object to be tracked among multiple target objects in response to the target selection instruction, and determine the state information of the target tracking object in the image frame, wherein the state information includes: speed and position; The target tracking module is used to determine the motion trajectory of the target object according to the state information of the target tracking object, and send a control instruction to the camera according to the motion trajectory, wherein the control instruction is used to instruct the camera to adjust the shooting angle and direction.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the camera monitoring method according to any one of claims 1 to 7 by running the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the camera monitoring method according to any one of claims 1 to 7 are implemented.

Images