Court trial violation behavior detection method and device, equipment, storage medium and product

Abstract

The invention provides a court trial violation behavior detection method and device, equipment, a storage medium and a product, and relates to the technical field of image processing, and the method comprises the steps: determining a first target region image of an image frame in a court trial video; according to the starting time point and the ending time point of court trial violation behavior detection, a second target area image is obtained from the first target area image of the court trial video; determining a human head, a display and a table card in the second target area image, and determining a target person and identity information in the second target area image based on the human head, the display and the table card in the second target area image; and performing target person identification on the second target area image based on the identity information to obtain a third target area image, and performing court trial violation behavior detection on the target person in the third target area image by applying a behavior analysis model. According to the invention, the comprehensiveness and objectivity of court hearing supervision are improved, and intelligent and refined supervision of the court hearing process is realized.

Description

Technical Field

[0001] The present invention relates to the field of image technology, and in particular to a method, device, equipment, storage medium and product for detecting courtroom violation behavior. Background Art

[0002] At present, the detection of courtroom violations mainly relies on manual spot checks of courtroom videos or real-time courtroom trials. This method is not only time-consuming and laborious, but also easily limited by the personal experience and subjective judgment of the spot checkers, resulting in randomness and incompleteness of supervision. Manual spot checks are difficult to cover all courtroom trials and may miss important courtroom violations. Summary of the invention

[0003] The present invention provides a method, device, equipment, storage medium and product for detecting court trial violations, so as to solve the problem that the detection of court trial violations in the prior art mainly relies on manual spot checks of court trial videos or real-time court trials. This method is not only time-consuming and labor-intensive, but also easily limited by the personal experience and subjective judgment of the spot checkers, resulting in the defects of randomness and incompleteness of supervision.

[0004] The present invention provides a method for detecting court trial violations, the method comprising: Acquire a court trial video, where the court trial video includes a plurality of consecutive image frames; For each image frame in the court trial video, determining a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; Determine the starting time point and the ending time point of the court trial violation detection, and select a plurality of second target area images between the starting time point and the ending time point from the plurality of first target area images of the court trial video; Determine the human head, display and table name in each of the second target area images, and determine the target person and the identity information of the target person in each of the second target area images based on the human head, display and table name in the second target area images; The target person in the second target area image is identified based on the identity information of the target person in the second target area image to obtain a third target area image, and a behavior analysis model is applied to detect court trial violation behavior of the target person in the third target area image, wherein the behavior analysis model is obtained by training a YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the person behavior corresponding to the first behavior label is a court trial violation behavior, and the person behavior corresponding to the second behavior label is a normal behavior.

[0005] According to a method for detecting courtroom violations provided by the present invention, the step of determining a first target area image in the image frame includes: Input the image frame into a multi-screen segmentation model to determine the area range of each screen corresponding to the image frame, wherein the area range of each screen is the area range of the image under each camera viewing angle in the image frame; Based on the area range of each picture, the image frame is divided into a plurality of area images; At least one candidate area image including a court trial marker image is screened out from the multiple area images corresponding to the image frame, and the candidate area image with the largest image area among the at least one candidate area image is determined as the first target area image in the image frame.

[0006] According to a method for detecting court trial violations provided by the present invention, determining the start time point and the end time point of the court trial violation detection includes: Applying a sliding window to the plurality of first target area images to perform gavel-banging action detection, and obtaining gavel-banging action detection results under the plurality of sliding windows; Based on the gavel-banging action detection results under the multiple sliding windows, determining a first sliding window in which the gavel-banging action is detected for the first time and a second sliding window in which the gavel-banging action is detected for the last time; Determining a starting time point for courtroom violation detection based on the position of the first sliding window; Based on the position of the second sliding window, the end time point of the court trial violation detection is determined.

[0007] According to a method for detecting courtroom violations provided by the present invention, the method of determining a target person in each second target area image based on a human head, a display and a table sign in the second target area image includes: Determine the aspect ratio of the display and the aspect ratio of the table cards in the second target area image; Determine a first target and a second target in the second target area image, wherein the first target includes a first display or a first table card having an aspect ratio less than 1, and the second target includes a second display or a second table card having an aspect ratio not less than 1; In the case where the upper left corner horizontal coordinate of the first target is smaller than the horizontal midline horizontal coordinate of the second target area image, determining that the person corresponding to the human head whose upper left corner horizontal coordinate is smaller than the upper left corner horizontal coordinate of the first target is a non-target person; In the case where the upper right corner horizontal coordinate of the first target is greater than the horizontal midline horizontal coordinate of the second target area image, determining that the person corresponding to the human head whose upper right corner horizontal coordinate is greater than the upper right corner horizontal coordinate of the first target is a non-target person; Determine the maximum lower left corner ordinate corresponding to the first target and the maximum lower left corner ordinate corresponding to the second target; Determine that the person corresponding to the human head whose lower left corner ordinate is smaller than the maximum lower left corner ordinate corresponding to the first target or smaller than the maximum lower left corner ordinate corresponding to the second target is a non-target person; Determine that persons corresponding to other human heads except the human head of the non-target person in the second target area image are target persons; Among them, the coordinate system of the second target area image is a rectangular coordinate system established with the upper left corner of the second target area image as the origin, the X-axis of the rectangular coordinate system is positive from left to right, the Y-axis of the rectangular coordinate system is positive from top to bottom, and each unit length represents a pixel.

[0008] According to a method for detecting courtroom violations provided by the present invention, the method further includes: Performing text recognition on the second table card in the second target area image, extracting and determining text content displayed on the second table card, wherein the text content is used to represent identity information of the corresponding person; In the second target area image, a DBSCAN clustering algorithm is applied to perform cluster analysis on the target persons and the second table cards to determine all the second table cards in the same row as each target person; The identity information of the target person is determined based on the text content displayed on the second table card matching the target person, wherein the second table card matching the target person is the second table card with the shortest distance to the target person among all the second table cards in the same row as the target person, and the distance is not greater than a preset threshold.

[0009] According to a method for detecting courtroom violation behavior provided by the present invention, the application behavior analysis model performs courtroom violation behavior detection on the target person in the third target area image, including: Inputting the third target area image frame by frame into the behavior analysis model to obtain a behavior detection result of each target person in each of the third target area images output by the behavior analysis model; The behavior detection results of each target person in each of the third target area images are traversed, and when the behavior detection results of the first target person in a preset number of the third target area images are all court trial violations, it is determined that the first target person has committed a court trial violation.

[0010] The present invention also provides a device for detecting courtroom violation behavior, the device comprising: A first court trial violation detection module is used to obtain a court trial video, where the court trial video includes a plurality of consecutive image frames; A second court trial violation detection module, configured to determine, for each image frame in the court trial video, a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; A third court trial violation detection module is used to determine the start time point and the end time point of the court trial violation detection, and to select a plurality of second target area images between the start time point and the end time point from the plurality of first target area images of the court trial video; A fourth courtroom violation detection module is used to determine the human head, display and table name in each of the second target area images, and based on the human head, display and table name in the second target area image, determine the target person and the identity information of the target person in each of the second target area images; The fifth court trial violation detection module is used to identify the target person in the second target area image based on the identity information of the target person in the second target area image, obtain a third target area image, and apply a behavior analysis model to detect court trial violations for the target person in the third target area image, wherein the behavior analysis model is obtained by training the YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the personnel behavior corresponding to the first behavior label is a court trial violation, and the personnel behavior corresponding to the second behavior label is a normal behavior.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for detecting courtroom violations as described above is implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for detecting courtroom violations.

[0013] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for detecting courtroom violations.

[0014] The court trial violation detection method, device, equipment, storage medium and product provided by the present invention first segment a first target area image for court trial violation detection from a court trial video, then screen out a second target area image within a specific time range from the first target area image, then accurately identify the target person in the second target area image based on the human head, display and table card information, and mark the target person in the second target area image based on the identity information of the target person in the second target area image to obtain a third target area image, and finally apply a behavior analysis model to perform court trial violation detection on the target person in the third target area image, thereby improving the comprehensiveness and objectivity of court trial supervision, avoiding randomness caused by personal experience and subjective judgment, and realizing intelligent and refined supervision of the court trial process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the method for detecting courtroom violation behavior provided by the present invention; Figure 2 It is a structural schematic diagram of the court trial violation detection device provided by the present invention; Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Figure 1 FIG. 1 is a flow chart of a method for detecting courtroom violations provided by an embodiment of the present invention. Figure 1 As shown, the method includes: Step 110, obtaining a court trial video, wherein the court trial video includes a plurality of continuous image frames; Here, the trial video refers to the video of the trial process obtained from the video recording equipment at the trial site, which includes all activities occurring during the entire trial process, such as the behavior and speeches of various persons in the courtroom.

[0020] Step 120, for each image frame in the court trial video, determining a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; It should be understood that in the court trial video, not all people in all areas need to be checked for violations. Therefore, each frame of the court trial video is processed to identify and mark the areas that need to be checked for court trial violations, i.e., the first target area images. These areas usually include the areas where people whose court trial violations need to be supervised are located.

[0021] Step 130, determining the start time point and the end time point of the court trial violation detection, and screening out a plurality of second target area images between the start time point and the end time point from the plurality of first target area images of the court trial video; In this embodiment, in order to improve the detection effect, only the image frames within the trial time period are further analyzed. Specifically, the second target area images within this period are screened out through the start time point and end time point of the trial violation detection.

[0022] Step 140, determining a human head, a display, and a table card in each of the second target area images, and determining a target person and identity information of the target person in each of the second target area images based on the human head, the display, and the table card in the second target area images; Here, a more detailed analysis is performed on the screened second target area image. Specifically, the human head, display (such as a laptop screen) and table sign (a sign used to identify the identity of a person) in the second target area image are identified, so as to determine the target person and the identity information of the target person who needs to be detected for court violation behavior in each image frame based on this information.

[0023] Step 150, identifying the target person in the second target area image based on the identity of the target person in the second target area image to obtain a third target area image, and applying a behavior analysis model to detect court trial violation behavior for the target person in the third target area image, wherein the behavior analysis model is obtained by training a YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the person behavior corresponding to the first behavior label is a court trial violation behavior, and the person behavior corresponding to the second behavior label is a normal behavior.

[0024] Here, the behavior analysis model is a computer vision technology that can recognize and understand the behavior of people in videos or images.

[0025] In this embodiment, the behavior analysis model is obtained by training the YOLO (You Only Look Once) target detection model based on the sample data set. These sample data include a large amount of image data, each of which is marked with a corresponding behavior label to indicate whether the person's behavior shown in the image data is a violation or non-violation.

[0026] Specifically, the first behavior label corresponds to the behaviors that are clearly defined as violations during the trial, such as not wearing or taking off the trial uniform, using mobile phones, inappropriate gestures, etc. The second behavior label corresponds to the normal and non-violation behaviors during the trial, such as sitting quietly, taking notes, and listening to speeches.

[0027] In this embodiment, after determining the target person and the identity information of the target person in each second target area image, each target person in each second target area image is identified based on the identity information of the target person to ensure that the target person in each image is known, such as identifying the target person with identity information A as A, and identifying the target person with identity information B as B. After identity identification, a new set of images, namely the third target area images, is obtained. The third target area images contain the target persons, and these target persons have been marked or associated with specific identity information.

[0028] The behavior analysis model is then used to determine the behavior type by analyzing features such as the target person's posture, movement trajectory, and relative positions of body parts in the third target area image. If the target person's behavior matches the pattern marked as a violation during the training of the behavior analysis model, the behavior analysis model will mark the behavior as a violation and associate it with the corresponding target person.

[0029] The method for detecting court violation behaviors proposed in the present embodiment first segments a first target area image for detecting court violation behaviors from the court trial video, then selects a second target area image within a specific time range from the first target area image, then accurately identifies the target person in the second target area image based on the human head, display and table card information, and identifies the target person in the second target area image based on the identity information of the target person in the second target area image to obtain a third target area image, and finally applies a behavior analysis model to perform court violation behavior detection on the target person in the third target area image, thereby improving the comprehensiveness and objectivity of court trial supervision, avoiding randomness caused by personal experience and subjective judgment, and realizing intelligent and refined supervision of the court trial process.

[0030] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0031] In some embodiments, determining the first target area image in the image frame includes: Input the image frame into a multi-screen segmentation model to determine the area range of each screen corresponding to the image frame, wherein the area range of each screen is the area range of the image under each camera viewing angle in the image frame; Based on the area range of each picture, the image frame is divided into a plurality of area images; At least one candidate area image including a court trial marker image is screened out from the multiple area images corresponding to the image frame, and the candidate area image with the largest image area among the at least one candidate area image is determined as the first target area image in the image frame.

[0032] It should be noted that there are usually multiple cameras shooting from different angles at the trial scene, and the images shot by multiple cameras from different angles are spliced ​​into a complete image frame, that is, the image frame in the trial video. In this embodiment, the multi-screen segmentation model is used to identify and segment the images from different camera perspectives contained in the image frame.

[0033] Here, the multi-screen segmentation model analyzes the color, texture, edge and other features in the image frame to determine the boundary and area range of each screen. Specifically, the segmentation results output by the multi-screen segmentation model can be expressed in the form of coordinate points, each of which corresponds to a point on the screen boundary. By connecting these points, a closed polygonal area can be formed to represent the boundary of each screen.

[0034] Based on the range of each screen area, the image frame will be segmented into multiple independent area images, each of which corresponds to an image under a specific camera angle. Among the multiple segmented area images, at least one candidate area image containing a court trial marker is screened out. Here, the court trial marker is usually located in the center or a prominent position of the court.

[0035] Furthermore, among the screened candidate area images containing court trial markers, the candidate area image with the largest image area is further found and determined as the first target area image.

[0036] The court trial violation detection method proposed in this embodiment can accurately determine the first target area image in the image frame through the above steps, and provide accurate input data for subsequent court trial violation detection.

[0037] In some embodiments, determining the start time point and the end time point of the court trial violation detection includes: Applying a sliding window to the plurality of first target area images to perform gavel-banging action detection, and obtaining gavel-banging action detection results under the plurality of sliding windows; Based on the gavel-banging action detection results under the multiple sliding windows, determining a first sliding window in which the gavel-banging action is detected for the first time and a second sliding window in which the gavel-banging action is detected for the last time; Determining a starting time point for courtroom violation detection based on the position of the first sliding window; Based on the position of the second sliding window, the end time point of the court trial violation detection is determined.

[0038] In this embodiment, the size and step length of the sliding window are first set. Here, the size of the sliding window can be determined according to the time taken for a complete gavel-banging action and the frame rate of the video, while the step length determines the speed at which the sliding window moves on the trial video. The smaller the step length, the slower the sliding window moves and the more details are detected. The larger the step length, the faster the window moves and the less computation is done, but some details may be missed. Typically, the step length is half or less of the size of the sliding window.

[0039] For example, the gavel action usually lasts 0.5 seconds, and the frame rate of the video is 30 frames per second, so the size of the sliding window can be 15 frames of images, and the step size is 5 frames of images.

[0040] Specifically, for each sliding window, it is necessary to extract features of the image frame in the sliding window, such as color, texture, shape, etc. Then, a pre-trained classifier or recognition model is used to judge the extracted features to determine whether the sliding window contains a gavel-strike action.

[0041] After obtaining the gavel-banging action detection results under multiple sliding windows, traverse the gavel-banging action detection results of all sliding windows, and record whether the gavel-banging action is detected in each sliding window. Finally, find the first and last windows where the gavel-banging action is detected, and record their positions (such as the number of the first image frame in the sliding window, or the number of the last image frame). Convert the position of the first sliding window (such as the number of the first image frame in the first sliding window) to the actual time (such as the time corresponding to the first image frame in the first sliding window), and use it as the starting time point for the detection of court violation behavior. Similarly, convert the position of the second sliding window (such as the first number of the first image frame in the second sliding window) to the actual time (such as the time corresponding to the first image frame in the second sliding window), and use it as the end time point for the detection of court violation behavior.

[0042] The method for detecting court trial violations proposed in this embodiment uses the gavel-banging action as a sign of the start and end of the trial to accurately determine the start and end time points of the court trial violation detection.

[0043] In some embodiments, determining the target person in each second target area image based on the human head, display and table sign in the second target area image includes: Determine the aspect ratio of the display and the aspect ratio of the table cards in the second target area image; Determine a first target and a second target in the second target area image, wherein the first target includes a first display or a first table card having an aspect ratio less than 1, and the second target includes a second display or a second table card having an aspect ratio not less than 1; In the case where the upper left corner horizontal coordinate of the first target is smaller than the horizontal midline horizontal coordinate of the second target area image, determining that the person corresponding to the human head whose upper left corner horizontal coordinate is smaller than the upper left corner horizontal coordinate of the first target is a non-target person; In the case where the upper right corner horizontal coordinate of the first target is greater than the horizontal midline horizontal coordinate of the second target area image, determining that the person corresponding to the human head whose upper right corner horizontal coordinate is greater than the upper right corner horizontal coordinate of the first target is a non-target person; Determine the maximum lower left corner ordinate corresponding to the first target and the maximum lower left corner ordinate corresponding to the second target; Determine that the person corresponding to the human head whose lower left corner ordinate is smaller than the maximum lower left corner ordinate corresponding to the first target or smaller than the maximum lower left corner ordinate corresponding to the second target is a non-target person; Determine that persons corresponding to other human heads except the human head of the non-target person in the second target area image are target persons; Among them, the coordinate system of the second target area image is a rectangular coordinate system established with the upper left corner of the second target area image as the origin, the X-axis of the rectangular coordinate system is positive from left to right, the Y-axis of the rectangular coordinate system is positive from top to bottom, and each unit length represents a pixel.

[0044] In this embodiment, the aspect ratio of the display and the table card in the second target area image is first calculated, and the elements in the second target area image are divided into two categories according to the aspect ratio: the first target with an aspect ratio less than 1 (which may be the first display with an aspect ratio less than 1 or the first table card with an aspect ratio less than 1) and the second target with an aspect ratio not less than 1 (which may be the second display with an aspect ratio not less than 1 or the second table card with an aspect ratio not less than 1). Based on this, people in different directions are distinguished.

[0045] If the upper left corner horizontal coordinate of the first target is smaller than the horizontal midline horizontal coordinate of the second target area image, the person corresponding to the human head whose upper left corner horizontal coordinate is smaller than the upper left corner horizontal coordinate of the first target is excluded. If the upper right corner horizontal coordinate of the first target is larger than the horizontal midline horizontal coordinate of the second target area image, the person corresponding to the human head whose upper right corner horizontal coordinate is larger than the upper right corner horizontal coordinate of the first target is excluded.

[0046] Furthermore, the maximum lower left ordinate of the first target and the second target is found, and the persons corresponding to the human heads whose lower left ordinate is smaller than the maximum lower left ordinate corresponding to the first target or the second target are excluded. After all non-target persons are excluded, the persons corresponding to the remaining human heads are the target persons.

[0047] The method for detecting courtroom violation behavior proposed in this embodiment can accurately identify the target person from the second target area image through the above steps.

[0048] In some embodiments, the method further comprises: Performing text recognition on the second table card in the second target area image, extracting and determining text content displayed on the second table card, wherein the text content is used to represent identity information of the corresponding person; In the second target area image, a DBSCAN clustering algorithm is applied to perform cluster analysis on the target persons and the second table cards to determine all the second table cards in the same row as each target person; The identity information of the target person is determined based on the text content displayed on the second table card matching the target person, wherein the second table card matching the target person is the second table card with the shortest distance to the target person among all the second table cards in the same row as the target person, and the distance is not greater than a preset threshold.

[0049] Here, an OCR (Optical Character Recognition) algorithm may be used to perform text recognition on the second table card in the second target area image, and extract and determine text content displayed on the second table card, which usually includes position identity information of a person.

[0050] Next, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm is applied to cluster the target personnel and the second table cards. Specifically, the DBSCAN clustering algorithm identifies the relative position relationship between the target personnel and the second table cards in space, and based on the identified relative position relationship, the target personnel and the second table cards are divided into different clusters, each cluster representing a row of target personnel and the corresponding second table cards.

[0051] For each target person, find all the second table cards in the same row (i.e., the second table cards in the same cluster) and calculate the distance (e.g., Euclidean distance) between the target person and each second table card in the same row. Find the second table card with the smallest distance that is not greater than a preset threshold (e.g., 150 pixels) as the second table card that matches the target person. Then, the text content on this second table card is used as the identity information of the target person.

[0052] In some embodiments, the applied behavior analysis model performs courtroom violation behavior detection on the target person in the third target area image, including: Inputting the third target area image frame by frame into the behavior analysis model to obtain a behavior detection result of each target person in each of the third target area images output by the behavior analysis model; The behavior detection results of each target person in each of the third target area images are traversed, and when the behavior detection results of the first target person in a preset number of the third target area images are all court trial violations, it is determined that the first target person has committed a court trial violation.

[0053] In this embodiment, the third target area image (the image that has been associated with the target person through the identity identifier) ​​is input into the behavior analysis model frame by frame, and the behavior analysis model processes each frame of the input third target area image and outputs the behavior detection results of each target person. These results may include the category of behavior (such as court violation behavior, normal behavior, etc.) and the corresponding confidence level.

[0054] Then, the behavior detection results of each target person in each third target area image are traversed. If it is detected that the behavior detection results of the first target person in a preset number (for example, 10 frames) of the third target area images are all court trial violations (that is, the category of the behavior is court trial violations, and the corresponding confidence levels are greater than 0.5), it is determined that the first target person has committed court trial violations.

[0055] The method for detecting court trial violations proposed in this embodiment implements court trial violation detection on the target person in the third target area image by applying a behavior analysis model, thereby realizing intelligent and refined supervision of the court trial process.

[0056] Based on any of the above embodiments, the present invention also provides a device for detecting courtroom violations, such as Figure 2 As shown, the device comprises: A first court trial violation detection module 210, for acquiring a court trial video, wherein the court trial video includes a plurality of consecutive image frames; A second court trial violation detection module 220, for determining, for each image frame in the court trial video, a first target region image in the image frame, wherein the first target region image is a region image for court trial violation detection; The third court trial violation behavior detection module 230 is used to determine the start time point and the end time point of the court trial violation behavior detection, and filter out a plurality of second target area images between the start time point and the end time point from the plurality of first target area images of the court trial video; The fourth courtroom violation detection module 240 is used to determine the human head, display and table card in each of the second target area images, and determine the target person and the identity information of the target person in each of the second target area images based on the human head, display and table card in the second target area images; The fifth court trial violation detection module 250 is used to identify the target person in the second target area image based on the identity information of the target person in the second target area image, obtain a third target area image, and apply a behavior analysis model to detect court trial violations for the target person in the third target area image, wherein the behavior analysis model is obtained by training the YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the person behavior corresponding to the first behavior label is a court trial violation, and the person behavior corresponding to the second behavior label is a normal behavior.

[0057] The device provided by the embodiment of the present invention first segments a first target area image for detecting court trial violations from a court trial video, then screens out a second target area image within a specific time range from the first target area image, then accurately identifies a target person in the second target area image based on the human head, display and table card information, and identifies the target person in the second target area image based on the identity information of the target person in the second target area image to obtain a third target area image, and finally applies a behavior analysis model to detect court trial violations on the target person in the third target area image, thereby improving the comprehensiveness and objectivity of court trial supervision, avoiding randomness caused by personal experience and subjective judgment, and realizing intelligent and refined supervision of the court trial process.

[0058] The court trial violation detection device described in this embodiment and the court trial violation detection method provided by the present invention described above can be referenced to each other and will not be described in detail here.

[0059] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the trial violation detection method, which includes: Acquire a court trial video, where the court trial video includes a plurality of consecutive image frames; For each image frame in the court trial video, determining a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; Determine the starting time point and the ending time point of the court trial violation detection, and select a plurality of second target area images between the starting time point and the ending time point from the plurality of first target area images of the court trial video; Determine the human head, display and table name in each of the second target area images, and determine the target person and the identity information of the target person in each of the second target area images based on the human head, display and table name in the second target area images; The target person in the second target area image is identified based on the identity information of the target person in the second target area image to obtain a third target area image, and a behavior analysis model is applied to detect court trial violation behavior of the target person in the third target area image, wherein the behavior analysis model is obtained by training a YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the person behavior corresponding to the first behavior label is a court trial violation behavior, and the person behavior corresponding to the second behavior label is a normal behavior.

[0060] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0061] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the court trial violation detection method provided by the above methods, the method comprising: Acquire a court trial video, where the court trial video includes a plurality of consecutive image frames; For each image frame in the court trial video, determining a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; Determine the starting time point and the ending time point of the court trial violation detection, and select a plurality of second target area images between the starting time point and the ending time point from the plurality of first target area images of the court trial video; Determine the human head, display and table name in each of the second target area images, and determine the target person and the identity information of the target person in each of the second target area images based on the human head, display and table name in the second target area images; The target person in the second target area image is identified based on the identity information of the target person in the second target area image to obtain a third target area image, and a behavior analysis model is applied to detect court trial violation behavior of the target person in the third target area image, wherein the behavior analysis model is obtained by training a YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the person behavior corresponding to the first behavior label is a court trial violation behavior, and the person behavior corresponding to the second behavior label is a normal behavior.

[0062] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for detecting courtroom violations provided by the above methods is implemented, and the method comprises: Acquire a court trial video, where the court trial video includes a plurality of consecutive image frames; For each image frame in the court trial video, determining a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; Determine the starting time point and the ending time point of the court trial violation detection, and select a plurality of second target area images between the starting time point and the ending time point from the plurality of first target area images of the court trial video; Determine the human head, display and table name in each of the second target area images, and determine the target person and the identity information of the target person in each of the second target area images based on the human head, display and table name in the second target area images; The target person in the second target area image is identified based on the identity information of the target person in the second target area image to obtain a third target area image, and a behavior analysis model is applied to detect court trial violation behavior of the target person in the third target area image, wherein the behavior analysis model is obtained by training a YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the person behavior corresponding to the first behavior label is a court trial violation behavior, and the person behavior corresponding to the second behavior label is a normal behavior.

[0063] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical coding feature diagrams therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting courtroom violations, characterized in that: include: Acquire a court trial video, where the court trial video includes a plurality of consecutive image frames; For each image frame in the court trial video, determining a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; Determine the starting time point and the ending time point of the court trial violation detection, and select a plurality of second target area images between the starting time point and the ending time point from the plurality of first target area images of the court trial video; Determine the human head, display and table name in each of the second target area images, and determine the target person and the identity information of the target person in each of the second target area images based on the human head, display and table name in the second target area images; Identify the target person in the second target area image based on the identity information of the target person in the second target area image to obtain a third target area image, and use a behavior analysis model to detect courtroom violation behaviors of the target person in the third target area image; Among them, the behavior analysis model is obtained by training the YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label. The personnel behavior corresponding to the first behavior label belongs to court trial violations, and the personnel behavior corresponding to the second behavior label belongs to normal behavior.

2. The method for detecting courtroom violations according to claim 1, characterized in that: The determining of the first target area image in the image frame comprises: Input the image frame into a multi-screen segmentation model to determine the area range of each screen corresponding to the image frame, wherein the area range of each screen is the area range of the image under each camera viewing angle in the image frame; Based on the area range of each picture, the image frame is divided into a plurality of area images; At least one candidate area image including a court trial marker image is screened out from the multiple area images corresponding to the image frame, and the candidate area image with the largest image area among the at least one candidate area image is determined as the first target area image in the image frame.

3. The method for detecting courtroom violations according to claim 1, characterized in that: The starting time point and the ending time point of the court trial violation detection are determined, including: Applying a sliding window to the plurality of first target area images to perform gavel-banging action detection, and obtaining gavel-banging action detection results under the plurality of sliding windows; Based on the gavel-banging action detection results under the multiple sliding windows, determining a first sliding window in which the gavel-banging action is detected for the first time and a second sliding window in which the gavel-banging action is detected for the last time; Determining a starting time point for courtroom violation detection based on the position of the first sliding window; Based on the position of the second sliding window, the end time point of the court trial violation detection is determined.

4. The method for detecting courtroom violations according to claim 1, characterized in that: The step of determining the target person in each of the second target area images based on the human head, the display and the table card in the second target area image comprises: Determine the aspect ratio of the display and the aspect ratio of the table cards in the second target area image; Determine a first target and a second target in the second target area image, wherein the first target includes a first display or a first table card having an aspect ratio less than 1, and the second target includes a second display or a second table card having an aspect ratio not less than 1; In the case where the upper left corner horizontal coordinate of the first target is smaller than the horizontal midline horizontal coordinate of the second target area image, determining that the person corresponding to the human head whose upper left corner horizontal coordinate is smaller than the upper left corner horizontal coordinate of the first target is a non-target person; In the case where the upper right corner horizontal coordinate of the first target is greater than the horizontal midline horizontal coordinate of the second target area image, determining that the person corresponding to the human head whose upper right corner horizontal coordinate is greater than the upper right corner horizontal coordinate of the first target is a non-target person; Determine the maximum lower left corner ordinate corresponding to the first target and the maximum lower left corner ordinate corresponding to the second target; Determine that the person corresponding to the human head whose lower left corner ordinate is smaller than the maximum lower left corner ordinate corresponding to the first target or smaller than the maximum lower left corner ordinate corresponding to the second target is a non-target person; Determine that persons corresponding to other human heads except the human head of the non-target person in the second target area image are target persons; Among them, the coordinate system of the second target area image is a rectangular coordinate system established with the upper left corner of the second target area image as the origin, the X-axis of the rectangular coordinate system is positive from left to right, the Y-axis of the rectangular coordinate system is positive from top to bottom, and each unit length represents a pixel.

5. The method for detecting courtroom violations according to claim 4, characterized in that: The method further comprises: Performing text recognition on the second table card in the second target area image, extracting and determining text content displayed on the second table card, wherein the text content is used to represent identity information of the corresponding person; In the second target area image, a DBSCAN clustering algorithm is applied to perform cluster analysis on the target persons and the second table cards to determine all the second table cards in the same row as each target person; The identity information of the target person is determined based on the text content displayed on the second table card matching the target person, wherein the second table card matching the target person is the second table card with the shortest distance to the target person among all the second table cards in the same row as the target person, and the distance is not greater than a preset threshold.

6. The method for detecting courtroom violations according to claim 1, characterized in that: The applied behavior analysis model detects courtroom violation behaviors of the target person in the third target area image, including: Inputting the third target area image frame by frame into the behavior analysis model to obtain a behavior detection result of each target person in each of the third target area images output by the behavior analysis model; The behavior detection results of each target person in each of the third target area images are traversed, and when the behavior detection results of the first target person in a preset number of the third target area images are all court trial violations, it is determined that the first target person has committed a court trial violation.

7. A device for detecting courtroom violations, characterized in that: include: A first court trial violation detection module is used to obtain a court trial video, where the court trial video includes a plurality of consecutive image frames; A second court trial violation detection module, configured to determine, for each image frame in the court trial video, a first target area image in the image frame, wherein the first target area image is an area image used for court trial violation detection; A third court trial violation detection module is used to determine the start time point and the end time point of the court trial violation detection, and to select a plurality of second target area images between the start time point and the end time point from the plurality of first target area images of the court trial video; A fourth courtroom violation detection module is used to determine the human head, display and table name in each of the second target area images, and based on the human head, display and table name in the second target area image, determine the target person and the identity information of the target person in each of the second target area images; The fifth court trial violation detection module is used to identify the target person in the second target area image based on the identity information of the target person in the second target area image, obtain a third target area image, and apply a behavior analysis model to detect court trial violations for the target person in the third target area image, wherein the behavior analysis model is obtained by training the YOLO target detection model based on a sample data set, and the sample data set includes image data identifying a first behavior label and image data identifying a second behavior label, the personnel behavior corresponding to the first behavior label is a court trial violation, and the personnel behavior corresponding to the second behavior label is a normal behavior.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the court trial violation detection method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for detecting courtroom violations as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for detecting courtroom violations as described in any one of claims 1 to 6.

Images