Article carrying identification method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an article carrying identification method and device, electronic equipment and a storage medium. The article carrying identification method comprises the following steps: extracting a target image from a monitoring video stream according to a set frame extraction mode; acquiring personnel information of each person contained in the current frame of target image; for each person, based on the person information of the person, whether the person is in an in-out state or not is judged, and based on the current frame target image and a previous frame target image of the current frame target image, optical flow analysis is carried out, and whether the person is suspected to carry an article or not is judged; when it is judged that the person is in the in-out state and it is judged that the person is suspected to carry the article, whether the person really carries the article or not is judged through a multi-mode large model; and when it is judged that the person really carries the article, it is determined that the person carries the article in and out. According to the embodiment of the invention, the efficiency and timeliness of article carrying identification can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an object handling identification method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of science and technology, items in many important places (such as large instruments and equipment) are often very important and expensive. According to regulations, non-internal personnel are not allowed to operate, move, or carry them in and out. However, there are certain difficulties in the management of items in actual production.

[0003] In order to monitor and record the movement of items, it is a very important requirement to use surveillance video to effectively identify whether items are being moved in and out. Currently, the identification of whether items are being moved in and out depends on manual supervision through surveillance video. However, manual supervision has shortcomings such as inadequate supervision and delayed detection, so the accuracy and efficiency of manual supervision are low. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application propose an object handling identification method, device, electronic device and storage medium to solve the problems of low accuracy and efficiency of manual supervision.

[0005] According to one aspect of an embodiment of the present application, a method for identifying object handling is provided, the method comprising:

[0006] Extract the target image from the surveillance video stream according to the set frame extraction method;

[0007] Obtaining the personnel information of each person contained in the target image of the current frame;

[0008] For each person, based on the person information of the person, determine whether the person is in or out of the state, and, based on the current frame target image and the previous frame target image of the current frame target image, perform optical flow analysis to determine whether the person is suspected of carrying an item;

[0009] When it is determined that the person is in an entry / exit state and it is determined that the person is suspected of carrying items, a multimodal large model is used to determine whether the person is actually carrying items;

[0010] When it is determined that the person is actually carrying an item, it is determined that the person is carrying an item in or out.

[0011] According to another aspect of an embodiment of the present application, there is provided an article transport identification device, the device comprising:

[0012] The extraction module is used to extract the target image from the monitoring video stream according to the set frame extraction method;

[0013] An acquisition module is used to acquire the personnel information of each person contained in the target image of the current frame;

[0014] A first judgment module is used to judge, for each person, whether the person is in an entry or exit state based on the person's information;

[0015] A second judgment module is used to perform optical flow analysis based on the current frame target image and the previous frame target image of the current frame target image to judge whether the person is suspected of carrying an object;

[0016] A third judgment module is used to use the multimodal large model to judge whether the person is actually carrying items when the first judgment module judges that the person is in an entry or exit state and the second judgment module judges that the person is suspected of carrying items;

[0017] The determination module is used to determine that the person is carrying items in and out when the third determination module determines that the person is actually carrying items.

[0018] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium; when the computer program is executed by the processor, the processor executes the object handling identification method as described in any one of the above items.

[0019] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the object handling identification method as described in any one of the above items.

[0020] In the embodiment of the present application, the target image is automatically extracted from the surveillance video stream according to the set frame extraction method, and the personnel information of each person contained in the current frame target image is obtained. Based on the personnel information of the personnel, it can be judged whether the personnel is in the entry and exit state, and optical flow analysis is also performed based on the current frame target image and the previous frame target image of the current frame target image to judge whether the personnel is suspected of carrying items. When it is judged that the personnel is in the entry and exit state, and when it is judged that the personnel is suspected of carrying items, a multimodal large model is further used to judge whether the personnel is really carrying items; when it is judged that the personnel is really carrying items, it is determined that the personnel is carrying items in and out. It can be seen from this that in the embodiment of the present application, based on the surveillance video, the personnel's entry and exit status, whether they are suspected of carrying items, and whether they are really carrying items are analyzed in multiple aspects, and whether the personnel are carrying items in and out are comprehensively identified, thereby improving the accuracy of item handling identification, and automatic identification based on the surveillance video can improve the efficiency and timeliness of item handling identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the embodiment of the present application, the drawings required for use in the description of the embodiment of the present application will be briefly introduced below. Obviously, the drawings described below are only some drawings of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a flow chart of an article handling identification method according to an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of an electronic threshold according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of an electronic threshold and personnel positions in an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of another electronic threshold and personnel position in an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of an optical flow mask image according to an embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a human body mask image according to an embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a first image in an embodiment of the present application;

[0029] Figure 8 is a schematic diagram of a second image in an embodiment of the present application;

[0030] Fig. 9 is a schematic diagram of a third image in an embodiment of the present application;

[0031] Fig.10 is a flow chart of another method for identifying object handling according to an embodiment of the present application;

[0032] Fig.11 is a structural block diagram of an article handling identification device according to an embodiment of the present application;

[0033] Fig.12 is a structural block diagram of an electronic device according to an embodiment of the present application;

[0034] Fig.13 It is a structural block diagram of a computer-readable storage medium in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] In life, there are many kinds of items in many important places and they are valuable, so they need to be properly supervised. Taking the computer room as an example, due to the large number of computer rooms and lack of manpower, it is often very difficult to supervise and detect the movement of items in and out (moving in and out) in time. Therefore, it is necessary to perform intelligent analysis of the video stream inside the computer room to detect the movement of items in and out in time.

[0037] In the embodiment of the present application, it is considered that if the judgment of whether to carry items in and out is made based on the posture of the human body, it is not only easy to misidentify, such as the person only makes a similar action of carrying but does not really carry the items, but also easy to miss the recognition, such as the human postures of different people carrying items in and out are different, and it is impossible to cover all human postures. If the carrying action of the person is identified based on wearable RFID (Radio Frequency Identification, radio frequency identification), it is necessary to configure RFID equipment, and the scope of application is narrow. If the overall model specifically used to identify the carrying of items in and out is trained, a large number of relevant samples need to be collected, the time cost of sample collection is large and highly customized, the training cycle is long, and it cannot be quickly promoted. Therefore, in the embodiment of the present application, it is proposed to analyze the entry and exit status of the person, whether it is suspected of carrying items, and whether it is really carrying items based on the surveillance video, and comprehensively identify whether the person is carrying items in and out, without judging the posture of the human body, without configuring the RFID equipment, and without training the overall model specifically used to identify the carrying of items in and out, so as to improve the accuracy and efficiency of the identification of the carrying of items, and the scope of application is wider.

[0038] Reference Figure 1 , shows a flow chart of an article handling identification method in an embodiment of the present application.

[0039] like Figure 1 As shown, the object handling identification method may include the following steps:

[0040] Step 101, extracting a target image from a surveillance video stream according to a set frame extraction method.

[0041] In the embodiment of the present application, the video capture device can be installed at a suitable location so as to capture the surveillance video stream at the door where the object is located, so as to monitor whether there are people carrying objects in and out of the door. For example, the shooting angle of the camera device can be directly facing the door where the object is located, etc. Among them, the object can be any object that needs to be monitored whether it is carried in and out, and the object will not be completely blocked by people.

[0042] In the embodiment of the present application, the monitoring video stream is pulled according to the address of the monitoring video stream, and an image is extracted from the monitoring video stream according to the set frame extraction method, and the extracted image is used as the target image. Exemplarily, the frame extraction method can be to extract one frame of image every preset number of frames, and so on. The specific value of the preset number of frames can be set according to actual needs, and this embodiment does not limit this.

[0043] Step 102: Obtain the personnel information of each person contained in the target image of the current frame.

[0044] After each frame of the target image is extracted, the personnel information of each person contained in the extracted current frame of the target image is obtained.

[0045] Exemplarily, the image processing model can be used to detect and track people in the current frame target image to obtain the personnel information of each person contained in the current frame target image. This method uses the image processing model for processing, which can improve the accuracy and efficiency of detection. Of course, other applicable methods can also be used to obtain the personnel information of each person contained in the current frame target image, such as image human body detection algorithm.

[0046] Exemplarily, the image processing model may include a human segmentation model and a human tracking model. The current frame target image is input into the human segmentation model and the human tracking model respectively, and the human segmentation model may output the human detection frame information, human mask matrix, etc. of each person contained in the current frame target image; the human tracking model may output the human detection frame information, human tracking identification, etc. of each person contained in the current frame target image. Therefore, the person information may include but is not limited to human detection frame information, human mask matrix, human tracking identification, etc.

[0047] The human body detection frame information may include the coordinates of the upper left corner of the human body detection frame, the coordinates of the lower right corner of the human body detection frame; or the coordinates of the lower left corner of the human body detection frame, the coordinates of the upper right corner of the human body detection frame; or the coordinates of the center point of the human body detection frame, the width of the human body detection frame, the height of the human body detection frame, etc. In the human body mask matrix, the value 0 may be used to represent the non-human body area, and the value 1 or the value 255 may be used to represent the human body area. The human body tracking identifier may be represented by the values ​​1, 2, 3, ...

[0048] For example, the human mask matrix can be expressed as follows, where 0 represents non-human areas: [[0 0 0...0 0 0] [0 0 0...0 0 0] [0 0 0...0 0 0] ... [0 0 0...0 0 0] [0 0 0...0 0 0] [0 0 0...0 0 0]]

[0056] In the embodiment of the present application, in order to simplify the processing process, an open source human body segmentation model and human body tracking model can be used for processing.

[0057] In an embodiment of the present application, in order to further improve accuracy, a smaller number of sample data (the number of sample data is smaller than the number of sample data for training an overall model specifically used to identify objects being carried in and out) may be collected to train a human segmentation model and / or a human tracking model.

[0058] First sample data is collected, where the first sample data includes a first sample image, annotated human body detection frame information of the first sample image, annotated human body mask matrix of the first sample image, and annotated human body tracking identifier of the first sample image.

[0059] A human body segmentation model to be trained is trained using the first sample data, wherein the input of the human body segmentation model to be trained is the first sample image, and the output is the predicted human body detection frame information of the first sample image and the predicted human body mask matrix of the first sample image; a first loss function is calculated based on the labeled human body detection frame information of the first sample image, the labeled human body mask matrix of the first sample image, the predicted human body detection frame information of the first sample image, and the predicted human body mask matrix of the first sample image; when the first loss function is greater than or equal to a first preset threshold, it is determined that the training of the human body segmentation model to be trained is incomplete, and the parameters of the human body segmentation model to be trained are adjusted to continue training; when the first loss function is less than the first preset threshold, it is determined that the training of the human body segmentation model to be trained is completed, and a trained human body segmentation model is obtained.

[0060] A human body tracking model to be trained is trained using the first sample data, wherein the input of the human body tracking model to be trained is the first sample image, and the output is the predicted human body detection frame information of the first sample image and the predicted human body tracking identifier of the first sample image; a second loss function is calculated based on the labeled human body detection frame information of the first sample image, the labeled human body tracking identifier of the first sample image, the predicted human body detection frame information of the first sample image, and the predicted human body tracking identifier of the first sample image; when the second loss function is greater than or equal to a second preset threshold, it is determined that the training of the human body tracking model to be trained is incomplete, and the parameters of the human body tracking model to be trained are adjusted to continue the training; when the second loss function is less than the second preset threshold, it is determined that the training of the human body tracking model to be trained is completed, and a trained human body tracking model is obtained.

[0061] The human segmentation model to be trained and the human tracking model to be trained can adopt the YOLO (You Only Look Once) series models, such as YOLOV5 (YOLO version 5), YOLOV7 (YOLO version 7), YOLOV8 (YOLO version 8), etc. Taking YOLOV8 as an example, it supports a full range of visual AI (Artificial Intelligence) tasks, including detection, segmentation, posture estimation, tracking and classification, and is widely used in autonomous driving, video surveillance, medical imaging, smart homes and robots. The above-mentioned first loss function and second loss function include but are not limited to: cross entropy loss function, absolute value loss function, square loss function, mean square error loss function, and so on. The specific values ​​of the first preset threshold and the second preset threshold can be set according to actual needs, and this embodiment does not limit this.

[0062] After obtaining the personnel information of each person included in the current frame target image, the human tracking identifier, human detection frame information, and human mask matrix of each person can be recorded. For example, if the current frame target image contains 2 persons, the human tracking identifier can be recorded as track_ids={1, 2}, the human detection frame information can be recorded as boxes={1:..., 2:...} (the... in boxes represents the human detection frame information), and the human mask matrix can be recorded as masks={1:..., 2:...} (the... in masks represents the human mask matrix).

[0063] In the embodiment of the present application, it is also necessary to record the initial position coordinates, status labels, and alarm labels of each person included in the current frame target image. For example, if the current frame target image contains two people, the initial position coordinates can be recorded as paths = {1: ..., 2: ...} (the ... in paths represents the initial position coordinates), the status label can be recorded as status = {1: ..., 2: ...} (the ... in status represents the status label), and the alarm label can be recorded as alarm_labels = {1: ..., 2: ...} (the ... in alarm_labels represents the alarm label).

[0064] Exemplarily, for each person included in the target image of the current frame, it can be determined whether the person appears for the first time. For example, it can be determined whether the human body tracking mark of the person appears for the first time. If the human body tracking mark of the person appears for the first time, it is determined that the person appears for the first time. When the person appears for the first time, the current position coordinates of the person are obtained based on the person information of the person as the initial position coordinates of the person, the initial position coordinates of the person are recorded, and the status label of the person is recorded as unknown (unknown), and the alarm label of the person is recorded as no alarm (False).

[0065] Exemplarily, for each person included in the target image of the current frame, after determining that the person is carrying items in or out, the current position coordinates of the person are obtained based on the person's personal information as the initial position coordinates of the person, the initial position coordinates of the person are recorded, and the status label of the person is recorded as unknown (unknown); after the alarm, the alarm label of the person is recorded as alarmed (True).

[0066] Step 103, for each person, based on the person's personal information, determine whether the person is in or out of the state, and, based on the current frame target image and the previous frame target image of the current frame target image, perform optical flow analysis to determine whether the person is suspected of carrying items.

[0067] In the embodiment of the present application, there is no restriction on the order of the process of determining whether the person is in an entry or exit state and the process of determining whether the person is suspected of carrying items.

[0068] In an optional implementation, the process of determining whether the person is in the entry / exit state based on the person information of the person may include the following steps A1 to A3:

[0069] Step A1, obtaining the coordinates of the first endpoint of the set electronic threshold and the coordinates of the second endpoint of the electronic threshold.

[0070] In an embodiment of the present application, an electronic threshold can be set according to actual conditions such as the shooting angle of the camera device and the location of the doorway, and the first endpoint coordinates and the second endpoint coordinates of the electronic threshold can be set. The first segment point coordinates and the second endpoint coordinates refer to coordinates located in the target image.

[0071] Figure 2 Schematic diagram of an electronic threshold according to an embodiment of the present application. Figure 2 As shown, the coordinates of the first endpoint of the electronic threshold are C (x3, y3), and the coordinates of the second endpoint are D (x4, y4).

[0072] Step A2, obtaining the current position coordinates of the person based on the human body detection frame information of the person, and obtaining the recorded initial position coordinates of the person.

[0073] Exemplarily, the process of obtaining the current position coordinates of the person based on the person detection frame information may include: calculating the coordinates of the bottom midpoint of the person detection frame based on the person detection frame information as the current position coordinates of the person. Since the bottom midpoint of the person detection frame is close to the position of the person's feet, the bottom midpoint of the person detection frame can be used to more accurately represent the current position of the person.

[0074] For example, the human body detection frame information includes the coordinates of the upper left corner of the human body detection frame and the coordinates of the lower right corner of the human body detection frame. The horizontal coordinate of the midpoint of the bottom edge of the human body detection frame is half of the sum of the horizontal coordinates of the upper left corner of the human body detection frame and the horizontal coordinates of the lower right corner of the human body detection frame, and the vertical coordinate of the midpoint of the bottom edge of the human body detection frame is the vertical coordinate of the lower right corner of the human body detection frame.

[0075] Step A3, judging whether the person is in or out based on the first endpoint coordinates, the second endpoint coordinates, the current position coordinates and the initial position coordinates.

[0076] Exemplarily, the process of judging whether the person is in or out based on the first endpoint coordinates, the second endpoint coordinates, the current position coordinates, and the initial position coordinates may include: judging whether a first line segment formed by the first endpoint coordinates and the second endpoint coordinates intersects with a second line segment formed by the current position coordinates and the initial position coordinates; when the first line segment and the second line segment intersect, it indicates that the person has crossed the electronic threshold, and thus it can be determined that the person is in or out; when the first line segment and the second line segment do not intersect, it indicates that the person has not crossed the electronic threshold, and thus it can be determined that the person is not in or out. This method of judgment is more accurate and efficient.

[0077] The following is an example of a person in the target image of the current frame.

[0078] Figure 3 It is a schematic diagram of an electronic threshold and personnel positions in an embodiment of the present application.

[0079] The current position coordinates of the person are A(x1, y1)=(453, 495);

[0080] The initial position coordinates of the personnel are B(x2, y2) = (912, 709);

[0081] The coordinates of the first endpoint of the electronic threshold are C(x3, y3)=(415, 369);

[0082] The coordinates of the second end point of the electronic threshold are D(x4, y4)=(680, 369).

[0083] Determine whether the line segment AB formed by connecting the current position coordinates of the person and the initial position coordinates of the person intersects with the second line segment formed by connecting the first endpoint coordinates of the electronic threshold and the second endpoint coordinates of the electronic threshold.

[0084] For line segment AB and points C and D, calculate the cross product of vector AC and vector AB, and the cross product of vector AD and vector AB, respectively. Similarly, for line segment CD and points A and B, calculate the cross product of vector CA and vector CD, and the cross product of vector CB and vector CD, respectively.

[0085] The cross product of vector AC and vector AB is cross_AC_AB = (y2-y1) × (x3-x1) - (y3-y1) × (x2-x1) = (709-495) × (415-453) - (369-495) × (912-453) = 49702;

[0086] The cross product of vector AD and vector AB is cross_AD_AB = (y2-y1) × (x4-x1) - (y4-y1) × (x2-x1) = (709-495) × (680-453) - (369-495) × (912-453) = 106412;

[0087] The cross product of vector CA and vector CD is cross_CA_CD = (y4-y3)×(x1-x3)-(y1-y3)×(x4-x3)=(369-369)×(453-415)-(495-369)×(680-415)=-33390;

[0088] The cross product of vector CB and vector CD is cross_CB_CD = (y4-y3)×(x2-x3)-(y2-y3)×(x4-x3)=(369-369)×(912-415)-(709-369)×(680-415)=-90100.

[0089] The sign of the cross product can be used to determine the position of a point relative to a line segment. Specifically, if cross_AC_AB and cross_AD_AB have different signs, then point C and point D are on either side of line segment AB (or one is collinear and the other is on one side); if cross_AC_AB and cross_AD_AB have the same sign, then point C and point D are on the same side of line segment AB; if cross_CA_CD and cross_CB_CD have different signs, then point A and point B are on either side of line segment CD (or one is collinear and the other is on one side); if cross_CA_CD and cross_CB_CD have the same sign, then point A and point B are on the same side of line segment CD. Therefore, if cross_AC_AB×cross_AD_AB<0, and cross_CA_CD×cross_CB_CD<0, then line segment AB intersects line segment CD, otherwise line segment AB does not intersect line segment CD.

[0090] like Figure 3 As shown, after the above calculation, cross_AC_AB×cross_AD_AB>0, and cross_CA_CD×cross_CB_CD>0, so it can be determined that line segment AB and line segment CD do not intersect.

[0091] In an optional implementation, after determining that the person is in an entry or exit state, it is possible to further identify whether the person is in an entry state or a exit state based on the person's personal information, so as to more accurately identify the person's behavior of carrying items in and out.

[0092] Exemplarily, the process of identifying whether the person is in an entering state or a leaving state based on the person's personal information may include: obtaining the current position coordinates of the person based on the person's human body detection frame information, and obtaining the recorded initial position coordinates of the person; calculating the angle between the vector pointing from the initial position coordinates to the current position coordinates and the positive direction of the x-axis (the positive direction of the x-axis on the target image, that is, the horizontal right direction); when the angle is greater than 0 and less than 180°, determining that the person is in a leaving state; when the angle is greater than -180° and less than 0, determining that the person is in an entering state.

[0093] First, the difference between the initial position coordinates and the current position coordinates in the x-axis direction and the difference in the y-axis direction (that is, the x-axis component and the y-axis component of the vector pointing from the initial position coordinates to the current position coordinates) are calculated; then, based on the difference between the initial position coordinates and the current position coordinates in the x-axis direction and the difference in the y-axis direction, the angle between the vector pointing from the initial position coordinates to the current position coordinates and the positive direction of the x-axis is calculated.

[0094] For example, the current position coordinates are A(x1, y1) and the initial position coordinates are B(x2, y2).

[0095] The difference between the initial position coordinates and the current position coordinates in the x-axis direction is Δx=x1-x2.

[0096] The difference between the initial position coordinate and the current position coordinate in the y-axis direction is Δy = (frame.shape[0]-y1)-(frame.shape[0]-y2). Frame.shape[0]-y1, ​​frame.shape[0]-y2 are the vertical coordinate system of the target image (origin at the upper left corner) converted to the common mathematical coordinate system (origin at the lower left corner), and frame.shape[0] represents the height of the target image.

[0097] The angle θ between the vector BA and the positive direction of the x-axis is arctan2(Δy, Δx), and the unit is radians.

[0098] Then, θ is converted into an angle θ°=degrees(θ) to obtain the angle between the vector pointing from the initial position coordinate to the current position coordinate and the positive direction of the x-axis.

[0099] because Figure 3 The people shown are not in or out, so another example is used to illustrate this.

[0100] Figure 4 It is a schematic diagram of another electronic threshold and personnel positions according to an embodiment of the present application.

[0101] like Figure 4 As shown, the current position coordinates of the person are A(x1, y1)=(476, 326), and the initial position coordinates of the person are B(x2, y2)=(912, 11).

[0102] The difference between the initial position coordinates and the current position coordinates in the x-axis direction:

[0103] Δx=x1-x2=-436

[0104] The difference between the initial position coordinate and the current position coordinate in the y-axis direction:

[0105] Δy=(frame.shape[0]-y1)-(frame.shape[0]-y2)=-315

[0106] The angle between vector BA and the positive direction of the x-axis is θ = arctan2(Δy, Δx) = 2.516

[0107] θ converted to degrees θ° = degrees(θ) = 144.15°

[0108] It can be seen that θ° is 144.15°, which is greater than 0 and less than 180°, so the person is in the leaving state.

[0109] After determining whether the person is in or out, and specifically determining whether the person is in or out, the status label of the person recorded in status can be updated. If the person is out, the status label of the person in status is updated to "out"; if the person is in, the status label of the person in status is updated to "in"; if the person is not in or out, the status label of the person in status remains "unknown".

[0110] In an optional implementation, the process of performing optical flow analysis based on the current frame target image and the previous frame target image of the current frame target image to determine whether the person is suspected of carrying an item may include the following steps B1 to B4:

[0111] Step B1, using an optical flow estimation model to perform optical flow analysis on the current frame target image and the previous frame target image to obtain an optical flow matrix, obtaining an optical flow image based on the optical flow matrix, and obtaining an optical flow mask image corresponding to the optical flow image.

[0112] In an embodiment of the present application, an optical flow estimation algorithm is used to track all pixel points of the target image in the previous and next frames. The optical flow is mainly described by the motion information of the image brightness. This motion pattern refers to the obvious movement between an object, surface, edge and background under the same camera and the same viewing angle. This method evaluates the deformation between the two images.

[0113] Exemplarily, the current frame target image and the previous frame target image of the current frame target image are input into the optical flow estimation model, and the optical flow estimation model performs optical flow analysis on the current frame target image and the previous frame target image and outputs an optical flow matrix. The optical flow matrix contains the horizontal displacement information and vertical displacement information of each pixel point in the target image between the previous frame target image and the current frame target image, that is, the optical flow matrix is ​​the motion matrix of each pixel point in the horizontal direction and the vertical direction.

[0114] In the embodiment of the present application, the optical flow estimation model can adopt any applicable form of optical flow estimation model. For example, a dense optical flow model can be adopted. For example, a GMA (Learning to Estimate Hidden Motionswith Global Motion Aggregation) model is adopted. The GMA model is a technology used in the field of computer vision, especially in optical flow estimation tasks. The technology is based on the Transformer method and aims to solve the motion estimation problem of occluded objects by globally aggregating motion information.

[0115] Based on the optical flow matrix, an optical flow image is obtained in any applicable manner, such as a color coding method, an arrow diagram method, an offset diagram method, and the like.

[0116] Take the color coding method as an example. First, define the color wheel: define a color wheel according to the direction (angle) of the optical flow, and map different angles to different colors. Then calculate the color: for each pixel in the optical flow matrix, find the corresponding color on the color wheel according to the direction (angle) of its optical flow. Then adjust the saturation or brightness: adjust the saturation or brightness of the color according to the size (modulus) of the optical flow. The larger the optical flow, the brighter (or brighter) the color; the smaller the optical flow, the dimmer (or darker) the color. Finally, draw the image: assign the adjusted color to the corresponding pixel in the optical flow image, and finally get an intuitive optical flow image.

[0117] For example, after the optical flow matrix is ​​converted to color and drawn on a white background, an optical flow image is obtained. The areas with faster movement are darker and reddish in color, while the areas with slower movement are lighter and bluish in color. The background image is white.

[0118] After acquiring the optical flow image, the optical flow image is grayed to obtain a grayscale image of the optical flow image, and then the grayscale image of the optical flow image is converted into a binary image of the optical flow image, and finally the binary image of the optical flow image is inverted to obtain an optical flow mask image, and the size of the optical flow mask image is the same as that of the target image.

[0119] Figure 5 is a schematic diagram of an optical flow mask image in an embodiment of the present application. Figure 5 As shown, the white areas in the optical flow mask image represent people and objects in motion.

[0120] Step B2, obtaining a human body mask image corresponding to the current frame target image based on the human body mask matrix of each person.

[0121] First, a black background image of the same size as the target image is drawn, and then, according to the human mask matrix of each person, a human mask image corresponding to the target image of the current frame is drawn on the black background image. The values ​​representing the human area in the human mask matrix are represented as white in the human mask image, and the values ​​representing the non-human area in the human mask matrix are represented as black in the human mask image.

[0122] Figure 6 is a schematic diagram of a human body mask image according to an embodiment of the present application. Figure 6 As shown, the white areas in the human mask image represent the detected persons.

[0123] Step B3: Compare the optical flow mask image and the human body mask image to obtain the object area corresponding to the person.

[0124] Exemplarily, the process of obtaining the area of ​​the object corresponding to the person based on the comparison between the optical flow mask image and the human body mask image may include: enlarging the human body detection frame information of the person according to a preset ratio to obtain the enlarged human body detection frame information; intercepting a first image from the optical flow mask image according to the enlarged human body detection frame information, and intercepting a second image from the human body mask image according to the human body detection frame information of the person; performing difference calculation on the first image and the second image to obtain a third image; obtaining the area of ​​the contour contained in the third image, and calculating the sum of the areas of the contour as the area of ​​the object corresponding to the person.

[0125] Since the human body detection frame of the person is the area where the person is located, and the area of ​​the moving person and object in the optical flow mask image is larger than the human body detection frame of the person, it is necessary to enlarge the human body detection frame information of the person according to a preset ratio to obtain the enlarged human body detection frame information. The specific value of the preset ratio can be set according to actual needs, and this embodiment does not limit this.

[0126] The difference calculation of the first image and the second image specifically refers to the difference calculation of each corresponding pixel in the first image and the second image. Before the difference calculation, the first image and the second image may be expanded, eroded, etc. to improve the accuracy of the processing.

[0127] Using any applicable contour detection algorithm, contour detection is performed on the third image, the contour contained in the third image is obtained, and the area of ​​the contour is calculated. For example, contour detection can be performed using an image processing library such as OpenCV. The specific process of contour detection can be processed based on actual experience, and this embodiment will not be discussed in detail here.

[0128] For example, considering the possible existence of noise points, target contours with an area greater than the second preset area can be screened out from the contours contained in the third image, and the sum of the areas of the target contours is calculated as the area of ​​the item corresponding to the person. The specific value of the second preset area can be set according to actual needs, and this embodiment does not limit this. For example, the second preset area can be 800, 900, 1000, and so on.

[0129] Figure 7 is a schematic diagram of a first image in an embodiment of the present application. Figure 7 As shown, the white area in the first image represents the captured people and objects.

[0130] Figure 8 is a schematic diagram of a second image in an embodiment of the present application. Figure 8 As shown, the white area in the second image represents the captured person.

[0131] Fig. 9 is a schematic diagram of a third image in an embodiment of the present application. Fig. 9 As shown, the white area in the third image is obtained by subtracting the first image from the second image, and two contours, contour 1 and contour 2, can be detected in the third image.

[0132] Step B4: when the area of ​​the object is larger than a first preset area, determining that the person is suspected of carrying an object.

[0133] For example, when the area of ​​the item is larger than the first preset area, it can be determined that the person is suspected of carrying items. The second preset area is smaller than the first preset area. The specific value of the first preset area can be set according to actual needs, and this embodiment does not limit this. For example, the second preset area can be 2000, 2500, 2800, and so on.

[0134] In an optional implementation, when the area of ​​the item is larger than the first preset area, the human body detection frame information of the person can be enlarged according to a preset ratio to obtain the enlarged human body detection frame information; a fourth image can be captured from the current frame target image according to the enlarged human body detection frame information, and the fourth image can be stored in the image queue corresponding to the person; when the number of fourth images in the image queue is larger than a preset number, it is determined that the person is suspected of carrying items. In this way, when the number of fourth images meets the conditions, it is determined that the person is suspected of carrying items, thereby avoiding the problem of inaccurate final judgment results caused by a certain misjudgment. The specific value of the preset number can be set according to actual needs, and this embodiment does not limit this.

[0135] Step 104, when it is determined that the person is in an entry / exit state and it is determined that the person is suspected of carrying items, a multimodal large model is used to determine whether the person is actually carrying items.

[0136] When it is determined that the person is not in or out, and / or when it is determined that the person is not suspected of carrying any items, it can be determined that the person is not carrying any items in or out. It should be noted that when it is determined that the person is not in or out, it is not necessary to continue to determine whether the person is suspected of carrying any items; when it is determined that the person is not suspected of carrying any items, it is not necessary to continue to determine whether the person is in or out.

[0137] When it is determined that the person is in an entry or exit state (entering state or leaving state) and it is determined that the person is suspected of carrying items, the multimodal large model is used to determine whether the person is actually carrying items. If it is determined that the person is in an entry or exit state (entering state or leaving state) and it is determined that the person is suspected of carrying items, then it is very likely that the person is carrying items in and out. At this time, the multimodal large model is used for a second judgment, which can greatly reduce the judgment frequency of the multimodal large model and improve the analysis efficiency.

[0138] Exemplarily, the process of using the multimodal large model to determine whether the person is actually carrying items may include: enlarging the person's human body detection frame information according to a preset ratio to obtain the enlarged human body detection frame information; capturing a fourth image from the current frame target image based on the enlarged human body detection frame information; inputting the fourth image and preset prompt information (prompt) into the multimodal large model to obtain an output result of the multimodal large model, and the output result is used to indicate whether the person is actually carrying items.

[0139] In the embodiment of the present application, in order to simplify the processing process, an open source multimodal large model can be used for processing.

[0140] In an embodiment of the present application, in order to further improve the accuracy, a smaller number of sample data (the number of sample data is smaller than the number of sample data for training an overall model specifically used to identify the movement of items in and out) may be collected to train a large multimodal model.

[0141] The second sample data is collected, where the second sample data includes a second sample image and annotation information of the second sample image, where the annotation information is used to indicate whether the person in the second sample image actually carries an item.

[0142] The multimodal large model to be trained is trained using the second sample data, wherein the input of the multimodal large model to be trained is the second sample image and preset prompt information, and the output is prediction information of the second sample image, and the prediction information is used to indicate whether the person in the second sample image actually carries items; a third loss function is calculated based on the annotation information of the second sample image and the prediction information of the second sample image; when the third loss function is greater than or equal to a third preset threshold, it is determined that the training of the multimodal large model to be trained is not completed, and the parameters of the multimodal large model to be trained are adjusted to continue training; when the third loss function is less than the third preset threshold, it is determined that the training of the multimodal large model to be trained is completed, and a trained multimodal large model is obtained.

[0143] The above-mentioned multimodal large model to be trained can adopt the GLM (General Language Model) series models, such as GLM4V (GLM version 4), etc. GLM4V has powerful language understanding and generation capabilities, and can execute code, browse the Internet, draw pictures and other functions. The multimodal large model can also adopt the InternVL (Internal Multimodal Vision and Language) model, etc. The above-mentioned third loss function includes but is not limited to: cross entropy loss function, absolute value loss function, square loss function, mean square error loss function, etc. The specific value of the third preset threshold can be set according to actual needs, and this embodiment does not limit this.

[0144] Exemplarily, when the fourth image is stored in an image queue corresponding to the person, the process of using the multimodal large model to determine whether the person is actually carrying items may include: selecting a fourth image located in the middle area of ​​the queue from the image queue corresponding to the person as the target fourth image; inputting the target fourth image and preset prompt information into the multimodal large model to obtain an output result of the multimodal large model, wherein the output result is used to indicate whether the person is actually carrying items.

[0145] In the embodiment of the present application, considering that the target fourth image has a direct impact on the result of the secondary judgment of the multimodal large model, it is necessary to select a more appropriate target fourth image. Considering that if the first image is selected, the information such as the human body may be incomplete, making it difficult to judge the multimodal large model, and if the tail image is selected, the person is too close to the camera and may cause certain deformation, which will also affect the judgment effect of the multimodal large model. Therefore, the target fourth image located in the middle position can better express the information of the personnel transportation. Therefore, the fourth image located in the middle area of ​​the queue can be selected from the image queue corresponding to the personnel as the target fourth image. Of course, the fourth image located in other positions can also be selected as the target fourth image.

[0146] For example, if the total number of fourth images is an odd number, the fourth image with an index of (n-1) / 2 in the image queue can be selected as the target fourth image; if the total number of fourth images is an even number, the fourth image with an index of n / 2 in the image queue can be selected as the target fourth image, where n represents the total number of fourth images in the image queue.

[0147] For example, in the current frame target image, the entry and exit status of the person with human tracking identification 2 has been changed to "out", status = {1:'unknown', 2:'out'}, and at the same time, the image queue corresponding to the person is checked and it is found that the fourth image corresponding to the person with human tracking identification 2 has reached 10 frames, queue = {1:2,2:10}, then the person with human tracking identification 2 is likely to carry items in and out, so the fourth image with index 5 corresponding to the person with human tracking identification 2 in the queue is selected as the target fourth image.

[0148] The prompt information is used to guide the multimodal large model to analyze whether the input is actually carrying items. For example, the prompt information can be prompt: "Is the person in the picture carrying items?", etc. When the result returned by the multimodal large model is result = "Yes, the person in the picture is carrying a large orange ladder", it is considered that the person is carrying items; when the result returned by the multimodal large model is result = "No, the person in the picture is not carrying items", it is considered that the person is not carrying items.

[0149] Step 105, when it is determined that the person is actually carrying items, determining that the person is carrying items in and out.

[0150] When the multimodal large model determines that the person is actually carrying items, it can be determined that the person is carrying items in and out.

[0151] In the embodiment of the present application, after determining that the person is carrying items in and out, the alarm tag of the person can also be obtained. When the alarm tag is not an alarm, the alarm information for the person is output, and the alarm tag of the person is set to an alarm (True). The status tag of the person can also be restored to unknown (unknown), and the image queue corresponding to the person is cleared, and the current position coordinates of the person are obtained as the initial position coordinates of the person. Then continue to analyze the next frame of the target image in the above manner.

[0152] In the embodiment of the present application, based on the surveillance video, the entry and exit status of the personnel, whether they are suspected of carrying items, and whether they are actually carrying items are analyzed, and whether the personnel are carrying items in and out is comprehensively identified, thereby improving the accuracy of identifying the transportation of items, and realizing automatic identification based on the surveillance video, which can improve the efficiency and timeliness of identifying the transportation of items.

[0153] Reference Fig.10 , shows a flow chart of another method for identifying object handling according to an embodiment of the present application.

[0154] like Fig.10 As shown, the object handling identification method may include the following steps:

[0155] Step S1, extracting a target image from a surveillance video stream according to a set frame extraction method.

[0156] Step S2, calling the human segmentation model to obtain the human detection frame information and human mask matrix of each person contained in the current frame target image, calling the human tracking model to obtain the human tracking identification of each person contained in the current frame target image, and recording the relevant information of each person.

[0157] Among them, relevant information may include human tracking identification, human detection frame information, human mask matrix, initial position coordinates, status label, and warning label.

[0158] Step S3, determining whether the person is in or out of the state, and updating the state tag according to the determination result.

[0159] If the person is in the entering state, the state label is updated to "in"; if the person is in the leaving state, the state label is updated to "out"; if the person is not in the entering or leaving state, the state label is updated to "unknown";

[0160] Step S4, calling the optical flow estimation model to obtain the optical flow matrix, obtaining the optical flow mask image based on the optical flow matrix, obtaining the human body mask image based on the human body mask matrix of the person, and obtaining the object area corresponding to the person based on the comparison between the optical flow mask image and the human body mask image. When the object area is larger than the first preset area, intercepting the fourth image and storing it in the image queue corresponding to the person.

[0161] Step S5, when the person is in the entry or exit state and the number of fourth images in the image queue corresponding to the person is greater than a preset number, a fourth image located in the middle area of ​​the queue is selected from the image queue as the target fourth image.

[0162] Step S6, using the multimodal large model to determine whether the object is actually being transported based on the fourth target image; if so, triggering an alarm; if not, continuing to analyze the next frame.

[0163] Step S7, update the alarm tag of the person to alarmed, restore the status tag to unknown, clear the image queue corresponding to the person, reset the initial position coordinates of the person, and continue to analyze the next frame.

[0164] In the embodiment of the present application, the model judgment of combining personnel segmentation, personnel tracking, personnel entry and exit status judgment, and optical flow estimation is different from simple multi-target tracking. There is no need to define the category of items, and there is no need to collect a large number of training samples. The deployment of algorithm services can be quickly realized. The multimodal large model is combined for secondary judgment with high accuracy. The optical flow estimation is used to make a difference with the segmentation result, the false negative rate is low, the size of the item can be adjusted according to the actual situation, and it has a wide range of applications.

[0165] Reference Fig.11 , shows a structural block diagram of an article handling identification device in an embodiment of the present application.

[0166] like Fig.11 As shown, the article handling identification device may include the following modules:

[0167] The extraction module 1101 is used to extract the target image from the monitoring video stream according to the set frame extraction method;

[0168] The acquisition module 1102 is used to acquire the personnel information of each person contained in the current frame target image;

[0169] The first judgment module 1103 is used to judge, for each person, whether the person is in an entry or exit state based on the person information of the person;

[0170] The second judgment module 1104 is used to perform optical flow analysis based on the current frame target image and the previous frame target image of the current frame target image to judge whether the person is suspected of carrying an object;

[0171] The third judgment module 1105 is used to use the multimodal large model to judge whether the person is actually carrying items when the first judgment module judges that the person is in or out and the second judgment module judges that the person is suspected of carrying items;

[0172] The determination module 1106 is used to determine that the person is carrying items in and out when the third determination module determines that the person is actually carrying items.

[0173] Optionally, the acquisition module 1102 is specifically configured to perform personnel detection and personnel tracking on the current frame target image using an image processing model to obtain personnel information of each person contained in the current frame target image.

[0174] Optionally, the personnel information includes human body detection frame information; the first judgment module 1103 includes:

[0175] A first coordinate acquisition submodule, used to acquire the coordinates of a first endpoint of a set electronic threshold and the coordinates of a second endpoint of the electronic threshold;

[0176] A second coordinate acquisition submodule, used to acquire the current position coordinates of the person based on the human body detection frame information of the person, and acquire the recorded initial position coordinates of the person;

[0177] The state judgment submodule is used to judge whether the person is in the entry or exit state based on the first endpoint coordinates, the second endpoint coordinates, the current position coordinates and the initial position coordinates.

[0178] Optionally, the second coordinate acquisition submodule is specifically used to calculate the coordinates of the bottom midpoint of the person's human body detection frame based on the person's human body detection frame information as the current position coordinates of the person.

[0179] Optionally, the status judgment submodule is specifically used to determine whether a first line segment formed by the first endpoint coordinates and the second endpoint coordinates intersects with a second line segment formed by the current position coordinates and the initial position coordinates; when the first line segment and the second line segment intersect, it is determined that the person is in an entry or exit state.

[0180] Optionally, the device further includes: an identification module, configured to identify whether the person is in the entering state or the leaving state based on the person information of the person after the first determination module determines that the person is in the entering or leaving state.

[0181] Optionally, the personnel information includes human body detection frame information; and the recognition module includes:

[0182] A third coordinate acquisition submodule is used to acquire the current position coordinates of the person based on the human body detection frame information of the person, and acquire the recorded initial position coordinates of the person;

[0183] An angle calculation submodule, used to calculate the angle between the vector pointing from the initial position coordinate to the current position coordinate and the positive direction of the x-axis;

[0184] The state determination submodule is used to determine that the person is in a leaving state when the angle is greater than 0 and less than 180°; and to determine that the person is in an entering state when the angle is greater than -180° and less than 0.

[0185] Optionally, the device also includes: a first recording module, used to obtain the current position coordinates of the person based on the personnel information of the person when the person appears for the first time, as the initial position coordinates of the person, and record the initial position coordinates of the person; a second recording module, used to obtain the current position coordinates of the person based on the personnel information of the person, as the initial position coordinates of the person, and record the initial position coordinates of the person after the determination module determines that the person is carrying items in or out.

[0186] Optionally, the personnel information includes a human body mask matrix; the second judgment module 1104 includes:

[0187] An optical flow analysis submodule, configured to perform optical flow analysis on the current frame target image and the previous frame target image using an optical flow estimation model to obtain an optical flow matrix, obtain an optical flow image based on the optical flow matrix, and obtain an optical flow mask image corresponding to the optical flow image;

[0188] An image acquisition submodule, used for acquiring a human body mask image corresponding to the current frame target image based on the human body mask matrix of each person;

[0189] An image comparison submodule, used for comparing the optical flow mask image and the human body mask image to obtain the object area corresponding to the person;

[0190] The transport determination submodule is used to determine that the person is suspected of transporting an item when the area of ​​the item is larger than a first preset area.

[0191] Optionally, the personnel information also includes human body detection frame information; the image comparison submodule includes:

[0192] A detection frame magnification unit, used to magnify the human body detection frame information of the person according to a preset ratio to obtain the magnified human body detection frame information;

[0193] An image capture unit, configured to capture a first image from the optical flow mask image according to the enlarged human body detection frame information, and capture a second image from the human body mask image according to the human body detection frame information of the person;

[0194] an image calculation unit, configured to perform difference calculation on the first image and the second image to obtain a third image;

[0195] The area calculation unit is used to obtain the areas of the contours contained in the third image, and calculate the sum of the areas of the contours as the area of ​​the object corresponding to the person.

[0196] Optionally, the area calculation unit is specifically used to filter out target contours with an area greater than a second preset area from the contours, and calculate the sum of the areas of the target contours as the area of ​​the item corresponding to the person; the second preset area is smaller than the first preset area.

[0197] Optionally, the personnel information also includes human body detection frame information; the transport determination submodule is specifically used to, when the area of ​​the item is larger than a first preset area, amplify the human body detection frame information of the person according to a preset ratio to obtain the amplified human body detection frame information; according to the amplified human body detection frame information, capture a fourth image from the current frame target image, and store the fourth image in an image queue corresponding to the person; when the number of fourth images in the image queue is greater than a preset number, determine that the person is suspected of transporting items.

[0198] Optionally, the third judgment module 1105 is specifically used to enlarge the human body detection frame information of the person according to a preset ratio to obtain the enlarged human body detection frame information; capture a fourth image from the current frame target image according to the enlarged human body detection frame information; input the fourth image and preset prompt information into the multimodal large model to obtain the output result of the multimodal large model, and the output result is used to indicate whether the person is actually carrying items.

[0199] Optionally, the third judgment module 1105 is specifically used to select a fourth image located in the middle area of ​​the queue from the image queue corresponding to the person as the target fourth image; input the target fourth image and preset prompt information into the multimodal large model to obtain an output result of the multimodal large model, and the output result is used to indicate whether the person is actually carrying items.

[0200] Optionally, the device also includes: an alarm module, which is used to obtain the alarm tag of the person after the determination module determines that the person is carrying items in and out, and when the alarm tag is not an alarm, output alarm information for the person and set the alarm tag of the person to an alarmed one.

[0201] In the embodiment of the present application, based on the surveillance video, the entry and exit status of the personnel, whether they are suspected of carrying items, and whether they are actually carrying items are analyzed, and whether the personnel are carrying items in and out is comprehensively identified, thereby improving the accuracy of identifying the transportation of items, and realizing automatic identification based on the surveillance video, which can improve the efficiency and timeliness of identifying the transportation of items.

[0202] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0203] In an embodiment of the present application, an electronic device is also provided. The electronic device may include a processor and a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium; when the computer program is executed by the processor, the processor executes the article handling identification method of any of the above embodiments.

[0204] Reference Fig.12 , shows a structural block diagram of an electronic device according to an embodiment of the present application. Fig.12 As shown, the electronic device 120 includes a processor 1201 and a computer-readable storage medium 1202 , on which a computer program 12021 is stored.

[0205] The processor 1201 is used to execute the computer program 12021 stored on the computer-readable storage medium 1202. When executing the computer program 12021, the processor 1201 implements the object handling identification method of any of the above-mentioned embodiments and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0206] The processor 1201 mentioned above may include but is not limited to: a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0207] The computer-readable storage medium 1202 mentioned above may include but is not limited to: read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), electronic erasable programmable read-only memory (EEPROM), hard disk, floppy disk, flash memory, etc.

[0208] In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. The computer program can be executed by a processor of an electronic device. When the computer program is executed by the processor, the processor executes the object handling identification method described in any of the above embodiments.

[0209] Reference Fig.13 , shows a block diagram of a computer-readable storage medium according to an embodiment of the present application. Fig.13 As shown, a computer program 1301 is stored on the computer-readable storage medium 130. When the computer program 1301 is executed by the processor, the processor executes the object handling identification method described in any of the above embodiments and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0210] The various embodiments in this specification are interrelated, and each embodiment is described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referenced to each other.

[0211] It should be noted that all actions to obtain signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the location and with the authorization given by the owner of the corresponding device.

[0212] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0213] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0214] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0215] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0216] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0217] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0218] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0220] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for identifying object handling, characterized in that: The method comprises: Extract the target image from the surveillance video stream according to the set frame extraction method; Obtaining the personnel information of each person contained in the target image of the current frame; For each person, based on the person information of the person, determine whether the person is in or out of the state, and, based on the current frame target image and the previous frame target image of the current frame target image, perform optical flow analysis to determine whether the person is suspected of carrying an item; When it is determined that the person is in an entry / exit state and it is determined that the person is suspected of carrying items, a multimodal large model is used to determine whether the person is actually carrying items; When it is determined that the person is actually carrying an item, it is determined that the person is carrying an item in or out.

2. The method according to claim 1, characterized in that: The obtaining of the personnel information of each person contained in the current frame target image includes: The image processing model is used to perform personnel detection and personnel tracking on the current frame target image to obtain personnel information of each person contained in the current frame target image.

3. The method according to claim 1, characterized in that The personnel information includes human body detection frame information; and judging whether the person is in or out of the state based on the personnel information of the person includes: Acquire the coordinates of the first endpoint of the set electronic threshold and the coordinates of the second endpoint of the electronic threshold; Acquire the current position coordinates of the person based on the human body detection frame information of the person, and acquire the recorded initial position coordinates of the person; Based on the first endpoint coordinates, the second endpoint coordinates, the current position coordinates and the initial position coordinates, it is determined whether the person is in an entry or exit state.

4. The method according to claim 3, characterized in that The obtaining the current position coordinates of the person based on the human body detection frame information of the person includes: Based on the human body detection frame information of the person, the coordinates of the bottom midpoint of the human body detection frame of the person are calculated as the current position coordinates of the person.

5. The method according to claim 3, characterized in that: The determining whether the person is in or out based on the first endpoint coordinates, the second endpoint coordinates, the current position coordinates, and the initial position coordinates includes: Determine whether a first line segment formed by connecting the first endpoint coordinates and the second endpoint coordinates intersects with a second line segment formed by connecting the current position coordinates and the initial position coordinates; When the first line segment intersects the second line segment, it is determined that the person is in an entry or exit state.

6. The method according to claim 1, characterized in that After determining whether the person is in the entry or exit state, the method further includes: After determining that the person is in the entry or exit state, based on the person information of the person, it is identified whether the person is in the entry state or the exit state.

7. The method according to claim 6, characterized in that The personnel information includes human body detection frame information; the identifying whether the person is in an entering state or a leaving state based on the personnel information of the person includes: Acquire the current position coordinates of the person based on the human body detection frame information of the person, and acquire the recorded initial position coordinates of the person; Calculate the angle between the vector pointing from the initial position coordinate to the current position coordinate and the positive direction of the x-axis; When the angle is greater than 0 and less than 180°, it is determined that the person is in a leaving state; When the angle is greater than -180° and less than 0, it is determined that the person is in the entry state.

8. The method according to claim 3 or 7, characterized in that: Before judging whether the person is in or out based on the person's information, the method further includes: when the person appears for the first time, obtaining the current position coordinates of the person based on the person's information as the initial position coordinates of the person, and recording the initial position coordinates of the person; After determining that the person carries the article in and out, the method further includes: acquiring the current position coordinates of the person based on the person information of the person as the initial position coordinates of the person, and recording the initial position coordinates of the person.

9. The method according to claim 1, characterized in that: The personnel information includes a human mask matrix; the optical flow analysis based on the current frame target image and the previous frame target image of the current frame target image is performed to determine whether the person is suspected of carrying an object, including: Performing optical flow analysis on the current frame target image and the previous frame target image using an optical flow estimation model to obtain an optical flow matrix, obtaining an optical flow image based on the optical flow matrix, and obtaining an optical flow mask image corresponding to the optical flow image; Acquire a human body mask image corresponding to the current frame target image based on the human body mask matrix of each of the persons; Comparing the optical flow mask image with the human body mask image, obtaining the area of ​​the object corresponding to the person; When the area of ​​the object is larger than the first preset area, it is determined that the person is suspected of carrying an object.

10. The method according to claim 9, characterized in that The personnel information also includes human body detection frame information; the area of ​​the object corresponding to the person is obtained by comparing the optical flow mask image with the human body mask image, including: Enlarging the human body detection frame information of the person according to a preset ratio to obtain enlarged human body detection frame information; intercepting a first image from the optical flow mask image according to the enlarged human body detection frame information, and intercepting a second image from the human body mask image according to the human body detection frame information of the person; performing difference calculation on the first image and the second image to obtain a third image; The areas of the contours contained in the third image are acquired, and the sum of the areas of the contours is calculated as the area of ​​the object corresponding to the person.

11. The method according to claim 10, characterized in that The calculating the sum of the areas of the contours as the area of ​​the object corresponding to the person includes: Target contours having an area greater than a second preset area are screened out from the contours, and the sum of the areas of the target contours is calculated as the area of ​​the item corresponding to the person; the second preset area is smaller than the first preset area.

12. The method according to claim 9, characterized in that The personnel information also includes human body detection frame information; when the area of ​​the object is larger than the first preset area, determining that the person is suspected of carrying an object includes: When the area of ​​the object is larger than the first preset area, the human body detection frame information of the person is enlarged according to a preset ratio to obtain the enlarged human body detection frame information; intercepting a fourth image from the current frame target image according to the enlarged human body detection frame information, and storing the fourth image in an image queue corresponding to the person; When the number of fourth images in the image queue is greater than a preset number, it is determined that the person is suspected of carrying an object.

13. The method according to claim 1, characterized in that The personnel information includes human body detection frame information; and the use of the multimodal large model to determine whether the personnel are actually carrying items includes: Enlarging the human body detection frame information of the person according to a preset ratio to obtain enlarged human body detection frame information; intercepting a fourth image from the current frame target image according to the enlarged human body detection frame information; The fourth image and preset prompt information are input into the multimodal large model to obtain an output result of the multimodal large model, and the output result is used to indicate whether the person actually carries the item.

14. The method according to claim 12, characterized in that The use of the multimodal large model to determine whether the person is actually carrying an item includes: Selecting a fourth image located in the middle area of ​​the queue from the image queue corresponding to the person as the target fourth image; The target fourth image and preset prompt information are input into the multimodal large model to obtain an output result of the multimodal large model, and the output result is used to indicate whether the person actually carries the item.

15. The method according to claim 1, characterized in that After determining that the person carries the article in and out, the method further includes: The alarm tag of the person is obtained, and when the alarm tag is not an alarm, the alarm information for the person is output, and the alarm tag of the person is set to an alarm.

16. An article handling identification device, characterized in that: The device comprises: The extraction module is used to extract the target image from the monitoring video stream according to the set frame extraction method; An acquisition module is used to acquire the personnel information of each person contained in the target image of the current frame; A first judgment module is used to judge, for each person, whether the person is in an entry or exit state based on the person's information; A second judgment module is used to perform optical flow analysis based on the current frame target image and the previous frame target image of the current frame target image to judge whether the person is suspected of carrying an object; A third judgment module is used to use the multimodal large model to judge whether the person is actually carrying items when the first judgment module judges that the person is in an entry or exit state and the second judgment module judges that the person is suspected of carrying items; The determination module is used to determine that the person is carrying items in and out when the third determination module determines that the person is actually carrying items.

17. An electronic device, characterized in that: The electronic device comprises a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program; When the computer program is executed by the processor, the processor is caused to execute the article transport identification method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the article handling identification method according to any one of claims 1 to 15.

Images