Service control method, device, equipment and medium

Abstract

The present application provides a service control method, device, equipment and medium, which relates to the field of artificial intelligence technology, and in particular to the field of image recognition technology. The method is used to improve the accuracy of service control. The method includes: obtaining a video clip to be processed containing a human body; identifying the movement trajectory of the target part of the human body in the video clip to be processed; obtaining a movement trajectory map corresponding to the movement trajectory according to the movement trajectory; performing content recognition on the movement trajectory map; and controlling the service according to the content recognition result. The method controls the service accordingly according to the recognition result of the movement trajectory generated by the target part of the human body during the movement. Compared with the method of recognizing the action of a single frame, the method can more accurately recognize the actual situation of the human body, thereby improving the accuracy of service control.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, in particular to image processing technology, and provides a service control method, device, equipment and medium. Background Art

[0002] With the continuous development of image processing technology, intelligent rescue technology has gradually emerged. The main functions of intelligent rescue technology are: identifying the real-time collected monitoring images, and determining whether the person in the monitoring image needs help based on the recognition results of the monitoring image. If the person needs help, an alarm is issued.

[0003] The existing intelligent rescue technology usually detects the action of a person in a single frame of surveillance images. If the action matches the pre-stored action, an alarm is issued. However, this detection method only alarms the action at a single moment, which is prone to false alarms. That is, the accuracy of the current intelligent rescue technology in processing services is low. Summary of the invention

[0004] The embodiments of the present application provide a service control method, apparatus, device and medium for improving the accuracy of service control.

[0005] On the one hand, a service control method is provided, comprising:

[0006] Obtain a video clip to be processed containing a human body;

[0007] Identify the movement trajectory of the target part of the human body in the video segment to be processed;

[0008] According to the movement trajectory, obtaining a movement trajectory map corresponding to the movement trajectory;

[0009] Performing content recognition on the movement trajectory graph;

[0010] Control services based on content recognition results.

[0011] The present application provides a service control device, including:

[0012] An acquisition module, used for acquiring a video clip to be processed containing a human body;

[0013] A trajectory recognition module, used to recognize the movement trajectory of the target part of the human body in the video segment to be processed;

[0014] An acquisition module, used for acquiring a movement trajectory map corresponding to the movement trajectory according to the movement trajectory;

[0015] A content recognition module, used for performing content recognition on the movement trajectory map;

[0016] The service control module is used to control the service according to the content recognition result.

[0017] In a possible embodiment, the trajectory recognition module is specifically used to:

[0018] For each video frame in the video segment to be processed, respectively determine the area of ​​the target part;

[0019] According to the area change of the target part in each video frame, the moving track of the target part in the video segment is determined.

[0020] In a possible embodiment, the trajectory recognition module is specifically used to:

[0021] For multiple video frames in the video clip to be processed, when it is identified that the action presented by the target part in the video frame is the first preset action in the preset behavior, start recording the relative position of the area of ​​the target part in the video frame until it is identified that the action presented by the target part in the video frame is the second preset action in the preset behavior, and obtain the movement trajectory.

[0022] In a possible embodiment, the relative position of the region of the target part in the video frame is the relative position of the target sub-part of the target part in the video frame.

[0023] In a possible embodiment, the target part includes multiple target parts in the human body, and the trajectory recognition module is specifically used to:

[0024] The regional changes of the multiple target parts in each video frame are spliced ​​to obtain a movement trajectory.

[0025] In a possible embodiment, the obtaining module is specifically used for:

[0026] The position of each point in the movement trajectory is mapped to a corresponding position in the image to obtain a movement trajectory map corresponding to the movement trajectory.

[0027] In a possible embodiment, the device further includes a display control module, and the display control module is used to:

[0028] After starting to record the relative position of the target part in the video frame, controlling to display the recorded relative position; and controlling to display the content recognition result of the movement trajectory diagram.

[0029] In a possible embodiment, the service control module is specifically used to:

[0030] When the content recognition result matches the preset content, an alarm message is generated and controlled to be displayed; the alarm message is used to indicate that the person corresponding to the human body is currently in a dangerous state.

[0031] In a possible embodiment, the video clip to be processed includes a plurality of continuous video frames including a human body detected from a video stream obtained by a surveillance camera; the alarm information includes the current location information of the human body; and the location information is obtained by any of the following methods:

[0032] Obtained by performing content recognition on any video frame in the video segment to be processed;

[0033] Determined according to the position of the surveillance camera.

[0034] An embodiment of the present application provides a computer device, including:

[0035] at least one processor, and

[0036] a memory communicatively coupled to the at least one processor;

[0037] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method as described in any one of the aspects by executing the instructions stored in the memory.

[0038] An embodiment of the present application provides a storage medium, wherein the storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method as described in any one of the aspects.

[0039] Since the embodiment of the present application adopts the above technical solution, it has at least the following technical effects:

[0040] In the embodiment of the present application, various image processing technologies are combined to determine the moving trajectory of the target part of the human body in the video clip, and the moving trajectory is converted into a moving trajectory map, and the content recognition is performed on the moving trajectory map to obtain the content recognition result, and the business control is performed according to the content recognition result. Since the business control is performed based on the recognition result of the moving trajectory of the internal position of the duration, the detailed information of the human body parts and the timing information in the process of seeking help are fully utilized, so that the misidentification caused by the single-frame video frame can be avoided, and the accuracy of business control is improved. Moreover, since the situation of mishandling is reduced, the situation of false alarm is naturally reduced, which reduces the burden on the staff. Moreover, the embodiment of the present application can be applied to the monitoring scene for all-round real-time rescue monitoring. When an emergency occurs, it can provide corresponding rescue in a timely, accurate and efficient manner, and monitor the emergency rescue by means of automatic monitoring cameras, reducing the workload of security personnel and improving the response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of an application scenario of a service control method provided in an embodiment of the present application;

[0042] Figure 2 A flow chart of a service control method provided in an embodiment of the present application Figure 1 ;

[0043] Figure 3 An example diagram of multiple video frames in a video clip to be processed provided in an embodiment of the present application;

[0044] Figure 4 An example diagram of a process for detecting a target part provided in an embodiment of the present application;

[0045] Figure 5 An example diagram of a movement trajectory provided in an embodiment of the present application;

[0046] Figure 6 An example diagram of a process of splicing the movement trajectories of various target parts provided in an embodiment of the present application;

[0047] Figure 7 An example diagram of a process of performing content recognition on a movement trajectory diagram provided in an embodiment of the present application;

[0048] Figure 8 Provided for the embodiments of this application Figure 1 Example diagram of interaction between terminal and server;

[0049] Fig. 9 A flow chart of a service control method provided in an embodiment of the present application Figure 2 ;

[0050] Fig.10 An example diagram of the process of detecting a hand provided in an embodiment of the present application;

[0051] Fig.11 An example diagram of detecting various gestures provided in an embodiment of the present application;

[0052] Fig.12 An example diagram of the process of character recognition on a movement trajectory diagram provided in an embodiment of the present application;

[0053] Fig.13 An example diagram of a terminal displaying alarm information provided in an embodiment of the present application;

[0054] Fig.14 An example diagram of the structure of a service control device provided in an embodiment of the present application;

[0055] Fig.15This is an example diagram of the structure of a computer control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to better understand the technical solution provided by the embodiments of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] In order to help those skilled in the art better understand the technical solution of the present application, the terms involved in the present application are introduced below.

[0058] 1. Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0059] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0060] 2. Computer Vision Technology (CV): Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0061] 3. Human key points: It means that if there is a human body in the detection image, the key points of the key parts of the human body can be detected. These human key points can describe the shape of the human body. When identifying the human body, 21 key points in the human body can be used for identification. These 21 key points are distributed in the main joints such as the top of the head, facial features, neck, limbs, etc.

[0062] 4. Part Affinity Field (PAF): A technology used to detect key points of the human body. The basic principle is to establish a directed field between two adjacent key points in the image, such as the left wrist and the left elbow. A bipartite graph is established for all the left wrists and left elbows found based on the field. Left and right matching is performed based on the bipartite graph. Successful matches are determined to be joints of the same person. Similarly, such matching operations are performed on all adjacent points to obtain the key points of each human body in the image.

[0063] 5. Target part: refers to a specific part of the human body. The target part can be a specific part of the human body, or multiple specific parts of the human body. The target part can be one or more of the head, hands or feet.

[0064] 6. Part key points: refers to the key points corresponding to the target part, which can show the overall shape of the target part. Taking the target part as hand as an example, the hand usually includes 21 key points, which are distributed on each finger and palm.

[0065] 7. Target sub-part: refers to a part of the target part. The target sub-part may be composed of key points of some parts of the target part. For example, if the target part is the hand, then the target sub-part may be the index finger.

[0066] 8. Movement trajectory: It is used to indicate the regional change of the target part in each video frame, which can be further understood as the position change result of the target part in the video clip. When there is one target part, the movement trajectory is the movement trajectory of the target part. When there are multiple target parts, the movement trajectory can be the combined result of the movement trajectories of the multiple target parts. The shape of the movement trajectory is related to the movement of the target part, so the shape of the movement trajectory is uncertain, and the specific shape can be, for example, a character, a string or a pattern.

[0067] 9. Movement trajectory map: refers to the movement trajectory presented by the image as the carrier, which can be the result of mapping each point in the movement trajectory into the image. The movement trajectory map will inherit the shape of the movement trajectory, so the content presented by the movement trajectory map corresponds to the movement trajectory.

[0068] 10. Preset behavior: refers to the behavior pre-stored in the device for triggering the execution of a business. The preset behavior is a collection of multiple continuous actions. An example of a preset behavior is the behavior of a user writing "SOS" with his index finger. The first preset action and the preset second preset action belong to the preset actions in the preset behavior, where the time of the first preset action in the preset behavior can be earlier than the time of the second preset action in the preset behavior. For example, the first preset action is the start action in the preset behavior, and the second preset action is the end action in the preset behavior. The first preset action and the second preset action can be the same action or different actions, and this application does not make specific restrictions.

[0069] The following is an introduction to the relevant technologies:

[0070] In the related art, a single frame of the monitoring screen is usually analyzed in real time. Once a person is detected to be performing a certain action on the monitoring screen, an alarm is issued. For example, if a person holds his head with both hands in the monitoring screen, it is determined that the person is in a dangerous state, and an alarm is issued. However, if the analysis is based only on the monitoring screen at a single moment, there is a high possibility of generating false alarms, that is, the accuracy of processing services in this way is low.

[0071] In view of this, an embodiment of the present application provides a service control method. The design concept of the service control method in the embodiment of the present application is introduced below.

[0072] The service control method in the embodiment of the present application can be applied to various monitoring scenarios, for example, it can be used to detect and identify the help-seeking behavior of a person under a monitoring camera, such as "SOS" written by a person under a monitoring camera, and respond promptly after detecting the help-seeking behavior of the person. The embodiment of the present application can realize the retrieval and analysis of the behavior of a person in a video based on image processing technology and various AI vision algorithms, such as human body detection algorithms, target part tracking and image recognition algorithms, and detect in real time whether the person under the monitoring camera needs help.

[0073] Specifically, in the embodiment of the present application, a human body detection algorithm is used to continuously monitor the human body in the video stream of the surveillance camera. When the presence of a human body is detected in the video stream, the target part of the human body is tracked in real time, and the relative position of the target part obtained in each video frame is recorded to obtain the movement trajectory of the target part. The movement trajectory is mapped into an image to obtain a movement trajectory map, and the content of the movement trajectory map is identified, and corresponding business control is performed according to the obtained content identification results. For example, if it is determined according to the content identification results that the person has sought help, an alarm message is generated, and the alarm message can also be pushed to the terminal device of the relevant personnel, so that the relevant personnel can go to the location corresponding to the location of the surveillance camera in time to provide assistance to the person.

[0074] Based on the above design concept, the application scenario of the service control method of the embodiment of the present application is introduced below.

[0075] Please refer to Figure 1 , is an application scenario diagram of a service control method provided in an embodiment of the present application, and the application scenario includes a monitoring camera 110, a monitoring center 120, and a storage system 130. The monitoring camera 110 and the monitoring center 120 can communicate with each other, and the monitoring camera 110 and the monitoring center 120 can communicate with each other respectively with the storage system 130, and the communication method can be any wireless or wired communication, and the present application does not make any specific restrictions.

[0076] Among them, the type of surveillance camera 110 can be arbitrary, such as a ball camera or a gun camera, etc. The surveillance camera 110 can include one or more cameras, and each surveillance camera 110 can be installed in different places, such as indoor places or outdoor places, etc. Indoor places include elevators or rooms, and outdoor places include roads or parks, etc. Figure 1 In the figure, two surveillance cameras 110 are used as an example, and the number of surveillance cameras is not actually limited. The monitoring center 120 can be implemented by a device with image processing capabilities, such as a terminal or a server, and the terminal can be a computer or a portable computer. The server can be a physical server or a cloud server, which can be a single server or a server cluster. The storage system 130 can generally refer to a device with storage capabilities, such as a cloud server.

[0077] In a first possible scenario, the monitoring center 120 may be implemented through one or more terminals 121:

[0078] The monitoring camera 110 is used to collect video data within its monitoring range and send the video data to the monitoring center 120. In this case, the monitoring center 120 receives the video data sent by the monitoring camera 110, decodes the video data to obtain video frames, displays the video frames, and processes the video frames to determine whether corresponding service control is required. The content of the service control method involved will be introduced below.

[0079] In the second possible scenario, the monitoring center 120 is implemented through the terminal 121 and the server 122:

[0080] The monitoring camera 110 is used to collect video data within its monitoring range and send the video data to the monitoring center 120. The server 122 obtains the video data from the monitoring camera 110 or from the terminal 121, decodes the video data, thereby obtaining a video frame, sends the video frame to the terminal 121 for display, and processes the video frame to determine whether corresponding service control is required, and the content of the service control method involved will be introduced below. Furthermore, the service control result can also be sent to the terminal 121 in a timely manner so that the terminal 121 can display it in a timely manner.

[0081] The surveillance camera 110 may also directly send the video data to the storage system 130 for storage. Alternatively, after being processed by the terminal 121 or the server 122, the video data that meets the storage conditions may be sent to the storage system 130 for storage, and the storage system 130 obtains and stores the video data. The storage conditions may include, for example, video data that needs to be service controlled.

[0082] Of course, in a specific implementation, the monitoring center 120 may include the terminal 121 in the first possible application scenario, and may also include the server 122 and the terminal 121 in the second possible application scenario.

[0083] It should be noted that Figure 1 This is an example of the application scenario involved in the embodiments of the present application, and does not actually limit the application scenario of the service control method involved in the embodiments of the present application.

[0084] Based on the above Figure 1 The first possible scenario, the service control method of the embodiment of the present application is introduced below. The service control method is Figure 1 Terminal 121 in the execution, the following is combined Figure 2 The flowchart of the service control method shown is used to introduce the service control method:

[0085] S201, the terminal 121 obtains a video segment to be processed that includes a human body.

[0086] As discussed above, since the number of surveillance cameras 110 may be large, the monitoring center 120 may include multiple terminals 121. Therefore, in an embodiment of the present application, one terminal 121 can be connected to one or more fixed surveillance cameras 110. The so-called connection can be understood as the terminal 121 being used to process the surveillance images collected by the connected surveillance camera 110.

[0087] Alternatively, the multiple terminals 121 include a master terminal and multiple slave terminals. The master terminal is used to monitor the working status of other terminals and distribute the video data of the surveillance camera 110 to the corresponding slave terminals for processing. For example, the master terminal can distribute the surveillance images collected by the surveillance camera 110 to the slave terminals that are currently experiencing low processing capacity or are idle. This can avoid excessive processing capacity of a certain terminal 121 and relatively balance the business processing capacity of each slave terminal.

[0088] Before the monitoring camera 110 sends video data to the terminal 121, the monitoring camera 110 can detect the human body in the monitoring screen, encode multiple continuous monitoring screens containing the human body into a data stream, and send the data stream to the terminal 121, so that the terminal 121 can obtain the video clip to be processed containing the human body. In this embodiment, the monitoring camera 110 can preliminarily screen the monitoring screen, reduce the amount of data interaction with the terminal 121, and reduce the amount of data processing by the monitoring center 120.

[0089] Alternatively, the surveillance camera 110 directly encodes the collected surveillance images into a video stream and sends the video stream to the terminal 121. The terminal 121 receives and decodes the video stream to obtain multiple video frames, and performs human body detection on the video frames. Once a human body is detected, a corresponding plurality of continuous video frames containing the human body are obtained, which is equivalent to obtaining a video clip to be processed. In the embodiment of the present application, the surveillance camera 110 does not need to perform target detection, which reduces the processing amount of the surveillance camera 110.

[0090] When the monitoring camera 110 or the terminal 121 detects whether each video frame contains a human body, a target detection technology may be used to detect whether the video frame contains a human body. If the video frame does not contain a human body, the video frame is not subjected to subsequent processing. The target detection technology is, for example, Yolo or SDD detection technology.

[0091] Among them, Yolo detection technology uses convolutional networks to extract features and uses fully connected layers to predict features to detect whether there is a human body in the video frame. SDD (Single Shot MultiBox Detector) detection technology, SDD detection technology performs multiple convolutions on video frames to obtain multiple feature maps. The feature map contains the image information of the video frame, such as the grayscale value and contour features of the video frame, and locates and detects whether there is a human body in the video frame and the position of the human body in the video frame.

[0092] When specifically detecting a human body in a video frame, a video frame may include the human body of each of multiple people. Therefore, when the terminal 121 detects that a video frame includes multiple human bodies, a process corresponding to the number of human bodies can be created, and each process is used to process a video clip of a corresponding human body. In an embodiment of the present application, the terminal 121 can create a process according to the number of human bodies that need to be processed in the video frame, so as to realize parallel processing of different human bodies in the video clip and improve processing efficiency. The processing process for each human body is the same. The following is an example of processing a human body to introduce the service control method involved in the embodiment of the present application.

[0093] For example, see Figure 3 A plurality of video frames of the video segment to be processed are shown, Figure 3 Where (1) to (n) represent multiple video frames in the video segment to be processed. Figure 3 In (1) to (n), it can be seen that each video frame contains a human body 300. In addition, Figure 3 The arrows in the figure indicate the timeline sequence of the video clips to be processed, i.e. Figure 3 (1) to (n) represent multiple video frames from front to back along the time axis.

[0094] S202, the terminal 121 identifies the movement trajectory of the target part of the human body in the video segment to be processed.

[0095] In each of the multiple video frames, if there is a human body in the video frame, the target part in the human body can be identified. The terminal 121 can directly use image recognition to identify the area corresponding to the target part from the area of ​​the human body. For the convenience of describing the area of ​​the human body in the video frame, it can be called a human body area, and the area of ​​the target part in the video frame can be called a target part area. The terminal 121 can specifically extract the key points of the human body in the video frame, and identify the area of ​​the target part according to the key points of the human body. The key points of the human body can refer to the content discussed in the previous text, and will not be repeated here.

[0096] For example, terminal 121 can extract human key points based on OpenPose or HRNet technology. OpenPose technology processes video frames, determines possible joints in video frames, and identifies human bodies based on PAF. The Deep High-Resolution Representation Learning network (HRnet) starts with a high-resolution subnetwork as the first stage, gradually adds high-resolution to low-resolution subnetworks, forms more stages, and connects multi-resolution subnetworks in parallel, performs multiple multi-scale fusions, so that each high-resolution to low-resolution representation repeatedly receives information from other parallel representations, thereby obtaining rich high-resolution representations, thereby obtaining more accurate human key points.

[0097] After the terminal 121 obtains the key points of the human body, it can identify the area of ​​the target part according to the key points of the human body and the area range of the target part in the human body. The terminal 121 can pre-store the area range of the target part in the human body and determine the area corresponding to the target part according to the key points of the human body. Figure 3 For example, terminal 121 identifies Figure 3 The human body shown in (2) is obtained as follows Figure 4 The human body key points 410 are shown in FIG.

[0098] After identifying the target part in the video frame, the target part may move over time, so the relative position of the target part in the video frame will also change. Therefore, terminal 121 can track and detect multiple video frames to obtain the movement trajectory of the target part in the segment to be processed.

[0099] Specifically, terminal 121 can identify the relative position of the area of ​​the target part in each video frame, record the relative position of the area in multiple video frames, and thus obtain the movement trajectory of the target part. The movement trajectory can also be understood as the change of the area of ​​the target part, where the change of the area includes the change of the relative position of the area in the video frame.

[0100] In specific implementation, since the target part is a relatively large area in the video frame, the change of the target part may not be very obvious. Therefore, if the movement trajectory of the area is directly identified, the obtained movement trajectory may not be obvious. Therefore, in the embodiment of the present application, the target sub-part can be determined from the target part, and the regional change of the target sub-part is determined as the movement trajectory of the target part. The content of the target sub-part can refer to the content discussed above, and will not be repeated here.

[0101] For example, continue to use Figure 3-4For example, after detecting the key points of the human body, the terminal 121 detects the hand area and detects the key points 420 of the hand area. The terminal 121 further takes the target sub-part as the index finger as an example, and detects Figure 3 The hand position shown in (2) is "cloth", which is the starting position in the preset behavior. Figure 3 The video frame shown in (2) starts recording the relative position of the index finger until the detection Figure 3 The posture of the video frame shown in (n) is the end posture in the preset behavior, thereby obtaining the movement trajectory of the hand.

[0102] In an actual monitoring scenario, since there are various people, each person may have various actions under monitoring. If the terminal 121 performs a comprehensive detection on any movement trajectory of each person, the processing volume of the terminal 121 will be large. Therefore, in an embodiment of the present application, the terminal 121 can track the target part in the video frames that meet the action conditions in multiple video frames to obtain the movement trajectory corresponding to the target part.

[0103] Specifically, the terminal 121 can identify the action presented by the target part in each video frame. When the terminal 121 detects that the action of the target part in a certain video frame is the first preset action in the preset behavior, it starts to record the relative position of the area of ​​the target part in the video frame until the action of the target part in another video frame is identified as the second preset action in the preset behavior, and the movement trajectory of the target part is obtained. The action presented by the target part can be understood as the posture presented by the target part in the video frame.

[0104] In specific implementation, the terminal 121 can identify the area of ​​the target part from the video frame. The identification of the area of ​​the target part can refer to the content discussed above, which will not be repeated here. After obtaining the area of ​​the target part, the terminal 121 can match the area of ​​the target part with the areas corresponding to various pre-stored actions, and use the action with the greatest matching degree as the action corresponding to the area of ​​the target part, thereby obtaining the action corresponding to the target part in the video frame. The action in the embodiment of the present application can also be called a gesture.

[0105] Furthermore, in order to ensure the accuracy of recognition, the terminal 121 may first identify key points of each part in the area of ​​the target part, and identify the action corresponding to the target part in the video frame according to the distribution of the key points of each part.

[0106] There may be multiple preset behaviors pre-stored in the terminal 121. When the terminal 121 detects that the action corresponding to a certain video frame belongs to the first preset action in any preset behavior, it starts detection, and when it detects that the action corresponding to another video frame belongs to the second preset action in the preset behavior, it stops detection, which can reduce the processing amount of the terminal 121 and obtain a more targeted movement trajectory.

[0107] It should be noted that the terminal 121 may detect the motion of the target part in the video frame after detecting the human body in the video frame. Alternatively, the terminal 121 may detect the motion of the target part in each of the multiple video frames after detecting the human body in each of the multiple video frames. The present application does not limit the timing of detecting the motion of the target part.

[0108] When there is only one target part, the terminal 121 directly tracks the target part to obtain the moving trajectory of the target part. In some cases, when a person is in a dangerous state, a body part may be free to move some time ago, but the movement of the body part of the person may be restricted due to the threat of criminals some time later, so the person may use the movement of other parts to send a distress signal. Therefore, in the embodiment of the present application, the target part may include multiple parts. In this case, the terminal 121 can track each target part to obtain the moving trajectory corresponding to each target part, and the moving trajectories of multiple target parts can be spliced ​​as a moving trajectory. Splicing can be understood as splicing the moving trajectories of each part in chronological order, and can be further understood as overlapping the moving trajectories of each target part in chronological order. The terminal 121 can also directly select a moving trajectory from multiple moving trajectories, for example, the terminal 121 can select a moving trajectory with the richest content information from the moving trajectory, and the content information includes the shape of the moving trajectory, etc.

[0109] For example, continue to use Figure 3 For example, terminal 121 records the relative position of the hand in the video frames from (2) to (n), and obtains Figure 5 For example, taking the target parts including the hand and the head as an example, the movement trajectory of the hand is recorded as follows Figure 6 The "S" shown in (1) is based on the movement trajectory of the head. Figure 6 The "OS" shown in (2) is spliced ​​together to obtain the following Figure 6 "SOS" shown in the trajectory diagram in (3).

[0110] It should be noted that, in the process of recording the position of the target part, the real-time recorded position of the target part is equivalent to the real-time movement trajectory. The terminal 121 can also display the recorded target part in real time while recording, so that the staff can view it in real time and discover abnormalities of the monitored person as early as possible.

[0111] S203: The terminal 121 obtains a movement trajectory map corresponding to the movement trajectory according to the movement trajectory.

[0112] As discussed above, the movement trajectory actually includes the position of the target part at each time. Therefore, after obtaining the movement trajectory, it is equivalent to obtaining each point. Therefore, the terminal 121 can map the position of each point to the corresponding position in the image. The image here can be any image template, for example, the image can be a blank picture that does not contain any image information, so as to obtain the movement trajectory map corresponding to the movement trajectory. Alternatively, the terminal 121 can take a screenshot of the recorded movement trajectory to obtain the movement trajectory map.

[0113] In a specific implementation, the terminal 121 may map each point in the moving trajectory after detecting the moving trajectory of the preset behavior to obtain a moving trajectory diagram, or the terminal 121 may map the obtained relative position in the image in real time while recording the relative position of the target part. When the moving trajectory is obtained, the moving trajectory diagram is naturally obtained.

[0114] For example, continue to use Figure 3 For example, terminal 121 will Figure 5 The moving trajectory shown is mapped into the image, and the following is obtained: Figure 7 The moving trajectory diagram shown in (1) is identified to obtain the moving trajectory diagram shown in (1) of 7. Figure 7 The content recognition result shown in (2) is as follows: Figure 7 "SOS" shown in (2).

[0115] Since people are usually unable to write help-seeking information on paper or other media when they are in danger or other special situations, in an embodiment of the present application, the movement trajectory of the target part of the human body is identified and the movement trajectory is cleverly converted into an image, thereby capturing various types of information left by the person in the absence of a medium, which is conducive to timely and accurate detection of the information that the person wants to express and providing timely help.

[0116] S204: The terminal 121 performs content recognition on the movement trajectory graph.

[0117] The terminal 121 can perform content recognition on the movement trajectory map, thereby obtaining a content recognition result of the movement trajectory map. The content specifically includes one or more of characters, graphics, patterns, etc. The terminal 121 can perform content recognition on the movement trajectory map according to various pre-stored content recognition models. For example, the terminal 121 can recognize characters in the movement trajectory map through a character recognition model, thereby obtaining recognized characters. Further, after obtaining the content recognition result, the terminal 121 can display the content recognition result.

[0118] S205: Terminal 121 performs service control according to the content recognition result.

[0119] After obtaining the content recognition result, if the terminal 121 determines that the monitored person is in a specific state, such as a dangerous state, based on the content recognition result, the terminal 121 can match the content recognition result with the pre-stored preset content. If the content recognition result matches the preset content, the terminal 121 determines that the person is in a dangerous state. The terminal 121 can perform corresponding services, such as initiating an alarm. For example, the terminal 121 can generate and display an alarm message, which is used to indicate that the person is in a dangerous state. The terminal 121 can also send the alarm message to the terminal of the target person so that the target person can handle it in time. When sending the alarm message to the target person, it can be sent via SMS or other instant messaging messages.

[0120] After receiving the alarm information, the terminal 121 may display the alarm information and the content recognition result at the same time. For example, the terminal 121 may display a monitoring screen and display the alarm information and the content recognition result in a floating window in the monitoring screen.

[0121] In order to facilitate the staff to determine the person's location in time for rescue, in an embodiment of the present application, the terminal 121 can locate the position information of the surveillance camera 110, or can identify any video frame among multiple video frames, so as to obtain the person's location information, and carry the location information in the alarm information. In this way, after obtaining the alarm information, the staff can directly locate the person's location information and perform rescue in time.

[0122] If the terminal 121 determines that the person is not in a dangerous state based on the content recognition result, the alarm service may not be executed and the above process may be repeated.

[0123] Further, when the terminal 121 detects that the content recognition result corresponding to the video segment to be processed matches the preset content, the terminal 121 can send the video segment to be processed to the storage system 130, so that the storage system 130 can store the video segment to be processed to leave corresponding evidence.

[0124] It should be noted that the above embodiment is an example of processing a video. In actual application, terminal 121 needs to process each video segment containing a human body. The processing process can refer to the content discussed above and will not be repeated here.

[0125] exist Figure 2 In the embodiment shown, automatic detection and automatic alarm are realized for the video frames in the monitoring scene, which reduces the workload of the staff. In addition, since the embodiment of the present application combines the action of the target part of the person in each video frame in multiple consecutive video frames and the moving trajectory of the target part, it is determined whether the person needs help in the monitoring scene, and then the corresponding business is executed, which improves the accuracy of business control. Moreover, the embodiment of the present application breaks through the conventional cognition that the recognition action is usually believed to determine whether a person needs help. In the embodiment of the present application, the action in the video frame and the help signal revealed by the person through the moving trajectory of the target part are combined to execute the corresponding business, improve the accuracy of business control, and reduce the possibility of false alarms.

[0126] The following is based on Figure 1 The second possible application scenario in the present application is introduced to introduce the service control method involved in the embodiment of the present application, please refer to Figure 8 The interaction diagram shown in the figure specifically includes:

[0127] S801, the server 122 obtains a video segment to be processed from the surveillance camera 110.

[0128] The method of obtaining the video clip to be processed can refer to the content discussed above, which will not be repeated here. In the embodiment of the present application, obtaining the video clip to be processed from the surveillance camera 110 is taken as an example. In fact, the server 122 can also obtain the video clip to be processed from the terminal 121.

[0129] S802, the server 122 identifies the movement trajectory of the target part of the human body in the video segment to be processed.

[0130] The method of obtaining the moving trajectory can refer to the content discussed above, which will not be repeated here.

[0131] S803 , the server 122 sends the recorded movement trajectory to the terminal 121 .

[0132] S804, the terminal 121 displays the recorded movement trajectory.

[0133] S805: The server 122 obtains a movement trajectory map corresponding to the movement trajectory according to the movement trajectory.

[0134] The method of obtaining the moving trajectory map can refer to the content discussed above, which will not be repeated here.

[0135] S806: The server 122 performs content recognition on the movement trajectory graph.

[0136] The method of performing content identification can refer to the content discussed above and will not be repeated here.

[0137] S807 , the server 122 sends the content recognition result to the terminal 121 .

[0138] S808: The terminal 121 displays the content recognition result.

[0139] S809, the server 122 generates warning information according to the content recognition result.

[0140] S810 , the server 122 sends the alarm information to the terminal 121 .

[0141] S811, terminal 121 displays warning information.

[0142] Among them, S803-S804 are optional parts, S807-S808 are optional parts, and S810-S811 are optional parts.

[0143] Figure 8 The server 122 in can also be used to implement Figure 2 Any service control method executed by the terminal 121 will not be described in detail here.

[0144] exist Figure 8 In the embodiment shown, the server 122 also controls the execution of services by combining the moving trajectory of the target part of the human body in multiple consecutive frames, thereby improving the accuracy of service control. Moreover, the server 122 performs video frame processing and service control, and the terminal 121 only needs to perform corresponding display, which can relatively reduce the processing amount of the terminal 121, and the two work together to improve the efficiency of service control.

[0145] In order to more clearly illustrate the service control method in the embodiment of the present application, the following takes the target part as an example, combined with Fig. 9 The flowchart of the business control method shown is introduced as follows:

[0146] S901, the server 122 obtains a video segment to be processed, and detects gestures in the video frame.

[0147] The server 122 receives the video stream and decodes the video stream to obtain multiple video frames. The server 122 performs human body detection on each video frame and continuously and in real time performs hand location and gesture recognition on the detected human body.

[0148] For example, the server 122 obtains Fig.10The human body in the video frame (1) and the local enlarged image of the human body are as follows: Fig.10 As shown in (2), the following is detected: Fig.10 The key points of the human body shown in (3) can be specifically referred to Fig.10 The server 122 then obtains the following information based on the key points of the human body: Fig.10 The hand area shown in (5) can be referred to as the partial enlarged view of the hand area. Fig.10 As shown in (6).

[0149] The server 122 can specifically locate the hand region according to the key point coordinates of the human wrist and elbow positions. The bounding box calculation formula of the hand region is as follows:

[0150]

[0151]

[0152] x hand =x wrist ,y hand =y wrist

[0153] Among them, x is the horizontal coordinate of a key point of the human body, and y is the vertical coordinate of the key point of the human body. hand is the horizontal coordinate of the center position of the hand bounding box, y hand is the ordinate of the center position of the hand bounding box, w hand is the width of the hand bounding box, h hand is the height of the hand bounding box. elbow Indicates the horizontal coordinate of the elbow key point, y elbow Indicates the ordinate of the elbow key point, x wrist Indicates the horizontal coordinate of the wrist key point, y wrist Indicates the vertical coordinate of the wrist key point.

[0154] After locating the hand area, the key points of the hand are identified. For example, the positions of the key points of the hand area can be obtained by using algorithms such as openpose, and the gestures are recognized based on the position arrangement and angle relationship of the key points of each part.

[0155] For example, the distribution of key points of the hand can refer to Fig.11 As shown in (1), when the server 122 determines that all five fingers are straightened, the gesture is determined as follows Fig.11 The server 122 determines that the key points of the five fingers can be arranged into straight lines, and the angle between adjacent straight lines is less than 45 degrees, then the hand posture is determined to be a fist, as shown in (2). Fig.11As shown in (3); when the server 122 recognizes that the key points 1, 5, 6, 7, and 8 are roughly arranged in a straight line and the remaining four fingers are bent, the gesture can be recognized as the state of the index finger extending, as shown in FIG. Fig.11 As shown in (4).

[0156] S902: After detecting the start action, the server 122 performs gesture tracking.

[0157] Once server 122 detects the appearance of a preset starting gesture, such as "paper", "V" or "OK", it performs part key point detection on the hand making the gesture in each subsequent video frame, for example, tracking specific positions such as the tip of the index finger, the palm of the hand, etc.

[0158] S903, after the server 122 detects the end of the action, it obtains the movement trajectory diagram of the hand.

[0159] For example, the server 122 sets the gesture "cloth" as the start gesture and the end gesture. When the server 122 detects the appearance of the gesture "cloth", it starts to record the relative position of the hand until the next gesture "cloth" appears, obtains the movement trajectory of the hand, and then maps the movement trajectory of the hand in the image, thereby obtaining a movement trajectory map. In this process, the movement trajectory of the index finger tip can be recorded in the embodiment of the present application as the movement trajectory of the hand. Since people use the index finger tip to write, the clarity of the detected movement trajectory can be improved.

[0160] S904, the server 122 performs character recognition on the movement trajectory diagram to obtain a character recognition result.

[0161] The server 122 may perform character recognition using a character recognition model or optical character recognition (OCR) technology, and the character recognition model may use various network neural models.

[0162] For example, see Fig.12 , which is an example diagram of a character recognition process provided in an embodiment of the present application, and the hand movement trajectory diagram obtained by the server 122 is as follows Fig.12 As shown in (1), the moving trajectory diagram is input into Fig.12 The neural network shown in (2) is obtained as follows Fig.12 The character recognition result shown in (3) is specifically "SOS".

[0163] S905: If the character recognition result matches the preset character, the server 122 generates an alarm message.

[0164] When the server 122 detects that the character recognition result matches the preset character, it generates an alarm message. The alarm message may also carry the location information of the person. The method of obtaining the location information can refer to the content discussed above and will not be repeated here. Further, after generating the alarm message, the server 122 may send the alarm message to the terminal 121 so that the terminal 121 can display the alarm message. Fig.13 The interface shown in the figure includes the "Warning!!!" shown in the alarm information and the character recognition result. The character recognition result is as follows Fig.13 "SOS" shown.

[0165] exist Fig. 9 In the embodiment shown, the server 122 can control the execution of the service based on the movement trajectory of the hand in multiple video frames. Since people usually write with their hands, Fig. 9 In the embodiment shown in , by detecting the movement trajectory of the hand, it is more likely to detect the help signal left by the person in special circumstances, so as to provide help to the person in need in time.

[0166] Based on the same inventive concept, the embodiment of the present application provides a service control device, which can be used to implement the functions of the terminal 121 or the server 122 discussed above. Fig.14 , the device comprises:

[0167] An acquisition module 1401 is used to acquire a video segment to be processed that contains a human body;

[0168] A trajectory recognition module 1402 is used to recognize the movement trajectory of the target part of the human body in the video segment to be processed;

[0169] An obtaining module 1403 is used to obtain a movement trajectory map corresponding to the movement trajectory according to the movement trajectory;

[0170] A content recognition module 1404 is used to perform content recognition on the movement trajectory map;

[0171] The service control module 1405 is used to control the service according to the content recognition result.

[0172] In a possible embodiment, the trajectory identification module 1402 is specifically used for:

[0173] For each video frame in the video clip to be processed, respectively determine the area of ​​the target part;

[0174] According to the regional changes of the target part in each video frame, the moving trajectory of the target part in the video clip is determined.

[0175] In a possible embodiment, the trajectory identification module 1402 is specifically used for:

[0176] For multiple video frames in the video clip to be processed, when it is identified that the action presented by the target part in the video frame is the first preset action in the preset behavior, the relative position of the area of ​​the target part in the video frame begins to be recorded until it is identified that the action presented by the target part in the video frame is the second preset action in the preset behavior, and the movement trajectory is obtained.

[0177] In a possible embodiment, the relative position of the region of the target part in the video frame is the relative position of the target sub-part in the target part in the video frame.

[0178] In a possible embodiment, the target part includes multiple target parts in the human body, and the trajectory recognition module 1402 is specifically used for:

[0179] The regional changes of multiple target parts in each video frame are spliced ​​to obtain the movement trajectory.

[0180] In a possible embodiment, the obtaining module 1403 is specifically used for:

[0181] The position of each point in the moving trajectory is mapped to the corresponding position in the image to obtain a moving trajectory map corresponding to the moving trajectory.

[0182] In a possible embodiment, the device further includes a display control module 1406, and the display control module 1406 is used to:

[0183] After starting to record the relative position of the target part in the video frame, controlling to display the recorded relative position; and controlling to display the content recognition result of the movement trajectory diagram.

[0184] In a possible embodiment, the service control module 1405 is specifically used to:

[0185] When the content recognition result matches the preset content, an alarm message is generated and controlled to be displayed; the alarm message is used to indicate that the person corresponding to the human body is currently in a dangerous state.

[0186] In a possible embodiment, the video clip to be processed includes a plurality of continuous video frames including a human body detected from a video stream obtained by a surveillance camera; the alarm information includes the current position information of the human body; and the position information is obtained by any of the following methods:

[0187] Obtained by performing content recognition on any video frame in the video clip to be processed;

[0188] Determined by the location of the surveillance camera.

[0189] Fig.14 The device can also be used to execute any of the service control methods discussed above, which will not be described in detail here.

[0190] Based on the same inventive concept, the present application embodiment provides a computer device, please refer to Fig.15 , the computer device includes a processor 1501 and a memory 1502.

[0191] The processor 1501 may be a central processing unit (CPU) or a digital processing unit, etc. The specific connection medium between the memory 1502 and the processor 1501 is not limited in the embodiment of the present application. Fig.15 In the embodiment, the memory 1502 and the processor 1501 are connected via a bus 1503. The bus 1503 is connected to the processor 1501 via a bus 1503. Fig.15 The connections between other components are shown in bold lines, and are not intended to be limiting. Bus 1503 can be divided into address bus, data bus, control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0192] The memory 1502 may be a volatile memory, such as a random-access memory (RAM); the memory 1502 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 1502 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1502 may be a combination of the above memories.

[0193] The processor 1501 is used to execute any of the above service control methods when calling the computer program stored in the memory 1502, and can also be used to implement Fig.14 The functions of the device shown may also be used to implement the functions of the terminal or server discussed above.

[0194] Based on the same inventive concept, an embodiment of the present application provides a storage medium, wherein the storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes any of the above-mentioned service control methods.

[0195] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0196] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned service control method.

[0197] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0198] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0199] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A service control method, characterized in that: include: Obtain a video clip to be processed containing a human body; For multiple target parts of the human body, respectively identifying movement trajectories of the target parts in the video clip to be processed; splicing the movement trajectories of the plurality of target parts in chronological order, and obtaining a movement trajectory map corresponding to the spliced ​​movement trajectory according to the spliced ​​movement trajectory; Performing content recognition on the movement trajectory graph; Control the business based on the content identification results; Among them, the controlling of the service according to the content recognition result includes: when the content recognition result matches the preset content, generating and controlling the display of an alarm message; the alarm message is used to indicate that the person corresponding to the human body is currently in a dangerous state.

2. The method according to claim 1, characterized in that The step of identifying the moving trajectory of the target part in the video segment to be processed includes: For each video frame in the video segment to be processed, respectively determine the area of ​​the target part; According to the regional changes of the target part in each video frame, the moving trajectory of the target part in the video segment to be processed is determined.

3. The method according to claim 2, characterized in that The step of determining the moving trajectory of the target part in the video segment to be processed according to the regional change of the target part in each video frame includes: For multiple video frames in the video clip to be processed, when it is identified that the action presented by the target part in the video frame is the first preset action in the preset behavior, start recording the relative position of the area of ​​the target part in the video frame until it is identified that the action presented by the target part in the video frame is the second preset action in the preset behavior, and obtain the movement trajectory.

4. The method according to claim 3, characterized in that The relative position of the area of ​​the target part in the video frame is the relative position of the target sub-part in the target part in the video frame.

5. The method according to any one of claims 1 to 4, characterized in that The step of obtaining a movement trajectory map corresponding to the spliced ​​movement trajectory according to the spliced ​​movement trajectory includes: The position of each point in the movement trajectory is mapped to a corresponding position in the image to obtain a movement trajectory map corresponding to the movement trajectory.

6. The method according to claim 3, characterized in that After the relative position of the target area in the video frame is recorded, the method further comprises: controlling the display of the recorded relative position; and, The controlling of the service according to the content recognition result also includes: controlling the display of the content recognition result of the movement trajectory map.

7. The method according to claim 6, characterized in that The video clip to be processed includes a plurality of continuous video frames including a human body detected from a video stream obtained by a surveillance camera; the alarm information includes the current position information of the human body; the position information is obtained by any of the following methods: Obtained by performing content recognition on any video frame in the video segment to be processed; Determined according to the position of the surveillance camera.

8. A service control device, characterized in that: include: An acquisition module, used for acquiring a video clip to be processed containing a human body; A trajectory recognition module, for respectively identifying the movement trajectories of the target parts in the video clip to be processed for the multiple target parts of the human body; An acquisition module, used for splicing the movement trajectories of the plurality of target parts in chronological order, and obtaining a movement trajectory map corresponding to the spliced ​​movement trajectory according to the spliced ​​movement trajectory; A content recognition module, used for performing content recognition on the movement trajectory map; A service control module, used to control services according to content recognition results; The business control module is specifically used to: generate and control the display of warning information when the content recognition result matches the preset content; the warning information is used to indicate that the person corresponding to the human body is currently in a dangerous state.

9. The device according to claim 8, characterized in that The trajectory recognition module is specifically used for: For each video frame in the video segment to be processed, respectively determine the area of ​​the target part; According to the regional changes of the target part in each video frame, the moving trajectory of the target part in the video segment to be processed is determined.

10. A computer device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method as described in any one of claims 1 to 7 by executing the instructions stored in the memory.

11. A storage medium, characterized in that: The storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 7.

Images