Method and device for determining violation behavior, storage medium and electronic device

Abstract

The application discloses a method for determining a violation behavior, and the method comprises the following steps: acquiring image data collected by an image collection device in a preset time period for a target area, wherein the target area comprises an area below a target device, the image collection device is arranged at the bottom of the target device to collect images of the target area; processing a plurality of images included in the image data to obtain an alarm area corresponding to the plurality of images and a movement track of a first object in the target area; and determining whether the first object has a violation behavior according to a position distribution relationship between a last track position of the movement track and the alarm area, wherein the violation behavior represents an abnormal behavior associated with the target area.

Description

Technical Field

[0001] This application relates to the field of security monitoring technology, and more specifically, to a method and apparatus for determining violations, a storage medium, and an electronic device. Background Technology

[0002] Currently, in common scenarios such as manufacturing, industry, and transportation, cranes and mobile cranes are generally used for lifting and hoisting operations, which can effectively reduce heavy manual labor and improve work efficiency. However, while these lifting and hoisting machines bring the benefits of industrialization and mechanization, they also bring certain safety hazards. For example, if the lifting and hoisting machine detaches or derails, and there are workers illegally positioned under the boom, it can pose a life-threatening danger.

[0003] Traditional methods of supervision to address such safety hazards mainly involve assigning inspectors to the site to monitor and correct any violations. However, due to the complex factory environment and blind spots in the monitoring area, inspectors are unable to promptly detect violations or alert violators, resulting in low safety levels in the supervision process.

[0004] Currently, one monitoring and alarm solution exists that involves installing cameras on the side of lifting and hoisting equipment to determine if workers are violating regulations based on the collected video data. However, this solution suffers from limitations because the cameras have a fixed viewing angle and can only capture video footage from a fixed location. For scenarios where cranes like bridge cranes are moving on tracks, it cannot monitor the movement of workers, accurately identify violations, and promptly send alarm messages.

[0005] Therefore, in related technologies, when lifting and hoisting machinery moves along the track, the fixed camera area makes it impossible to monitor the movement of workers and accurately identify their violations.

[0006] There is currently no effective solution to the technical problem that, when lifting and hoisting machinery moves along the track, the fixed camera area makes it impossible to monitor the movement of workers and accurately identify their violations. Summary of the Invention

[0007] This application provides a method and apparatus, storage medium and electronic device for determining violations, to at least solve the technical problem in the related art that, when lifting and hoisting machinery moves along the track, the fixed camera area makes it impossible to monitor the movement of workers and accurately identify their violations.

[0008] According to one embodiment of this application, a method for determining violations is provided, comprising: acquiring image data collected by an image acquisition device within a preset time period for a target area, wherein the target area includes an area below the target device; the image acquisition device is disposed at the bottom of the target device to acquire images of the target area; processing multiple frames of images included in the image data to obtain alarm areas corresponding to the multiple frames of images and the movement trajectory of a first object within the target area; determining whether the first object has committed a violation based on the last trajectory position of the movement trajectory and the positional distribution relationship of the alarm areas, wherein the violation indicates abnormal behavior associated with the target area.

[0009] In an exemplary embodiment, processing the image data, which includes multiple frames of images, to obtain alarm regions corresponding to the multiple frames of images includes: dividing each frame of the multiple frames of images based on the geometric center of each frame to obtain divided multiple frames of images, each frame of the divided multiple frames of images having multiple division regions; sending a calibration prompt message containing the divided multiple frames of images to a second object, and receiving feedback information sent by the second object in response to the calibration prompt message, wherein the feedback information includes the result of the second object calibrating the region type of the multiple division regions of each frame of the divided multiple frames of images, wherein the region type includes: alarm region and non-alarm region; determining the division region indicated in the feedback information as an alarm region as the alarm region corresponding to the multiple frames of images.

[0010] In an exemplary embodiment, processing the image data, which includes multiple frames, to obtain the movement trajectory of a first object within the target area includes: acquiring the current frame image and other frame images preceding the current frame image from the multiple frames, and determining a mapping relationship between the current frame image and the other frame images; wherein the mapping relationship represents the mathematical relationship required to convert the coordinate position of the first object in the other frame images into a coordinate position in the coordinate system of the current frame image; acquiring the first coordinate position of the first object in the other frame images, and determining the second coordinate position corresponding to the first coordinate position in the coordinate system of the current frame image according to the mapping relationship; acquiring the third coordinate position of the first object in the coordinate system of the current frame image; and determining the movement trajectory based on multiple second coordinate positions and the third coordinate position.

[0011] In an exemplary embodiment, determining the movement trajectory based on a plurality of second coordinate positions and the third coordinate position includes: determining the sum of the following parameters: the third coordinate position, the position difference between the second coordinate position and the third coordinate position; determining a fourth coordinate position in the coordinate system of the current frame image based on the sum, wherein the fourth coordinate position is the position of a fifth object in the coordinate system of the current frame image; determining a first object feature when the fifth object is located at the fourth coordinate position, a second object feature corresponding to the first object being located at the second coordinate position, and a third object feature corresponding to the first object being located at the third coordinate position; if the feature overlap between the first object feature and the second object feature is greater than a preset value, and the feature overlap between the first object feature and the third object feature is greater than the preset value, determining that the fifth object is consistent with the first object, and determining the movement trajectory of the first object in the current frame image based on the second coordinate position, the third coordinate position, and the fourth coordinate position.

[0012] In an exemplary embodiment, the position difference between a plurality of second coordinate positions and the third coordinate position is determined by: determining, from the plurality of second coordinate positions, the second coordinate position corresponding to the first frame image of a plurality of other frame images, and the second coordinate position corresponding to the last frame image of a plurality of other frame images, and determining the position offset value between the second coordinate position corresponding to the first frame image of a plurality of other frame images and the second coordinate position corresponding to the last frame image of a plurality of other frame images; obtaining the number of frames of a plurality of other frame images, determining the ratio of the position offset to the number of frames as the target second coordinate position, and determining the position difference between the target second coordinate position and the third coordinate position as the position difference between the plurality of second coordinate positions and the third coordinate position.

[0013] In an exemplary embodiment, determining the mapping relationship between the current frame image and the other frame images includes: when it is determined that a third object exists in both the current frame image and the other frame images, obtaining a first image feature of the third object in the current frame image and a second image feature of the third object in the other frame images; calculating a transformation matrix between the first image feature and the second image feature according to a preset algorithm to obtain the mapping relationship between the first image feature and the second image feature.

[0014] In an exemplary embodiment, before processing the multiple frames of images included in the image data to obtain the movement trajectory of the first object within the target area, the method further includes: determining the first object from the multiple frames of images, wherein determining the first object from the multiple frames of images includes: acquiring different objects identified in the multiple frames of images; identifying the different objects based on their object features to obtain object types corresponding to the different objects; acquiring a fourth object among the different objects that has the same object type, and determining the fourth object as the first object if the identity of the fourth object is consistent with the identity type of the first object.

[0015] In an exemplary embodiment, during the process of acquiring different objects identified in the multi-frame images, the method further includes: for each of the different objects, determining a first object area preset for each object, and acquiring a second object area of ​​each object in the multi-frame images; if the difference between the first object area and the second object area is less than or equal to a preset value, determining the object feature corresponding to the first object area as the object feature of each object.

[0016] In an exemplary embodiment, determining whether the first object has engaged in a violation based on the relationship between the final trajectory position of the movement trajectory and the location distribution of the alarm area includes: obtaining a preset number of continuous trajectory points from all trajectory points included in the final trajectory position; obtaining each trajectory ray of each continuous trajectory point at a preset angle; if it is determined that each trajectory ray intersects with the boundary line of the alarm area, obtaining the number of intersections between each trajectory ray and the boundary line of the alarm area; if it is determined that the number of intersections between each trajectory ray and the boundary line of the alarm area satisfies a preset condition, and if the final trajectory position of the movement trajectory is located within the alarm area, then it is determined that the first object has engaged in a violation.

[0017] In an exemplary embodiment, determining whether the first object has committed a violation based on the relationship between the last trajectory position of the movement trajectory and the location distribution of the alarm area includes: obtaining a preset number of continuous trajectory points from all trajectory points included in the last trajectory position; obtaining any trajectory ray of any trajectory point of any of the continuous trajectory points at the preset angle; if it is determined that the number of intersections between any trajectory ray and the boundary line of the alarm area does not meet the preset condition, and the last trajectory position of the movement trajectory is not located within the alarm area, then it is determined that the first object has not committed a violation.

[0018] In an exemplary embodiment, after determining that the first object has committed a violation, the method further includes: sending an alarm message to the first object if the last trajectory position is located within the alarm area.

[0019] According to another embodiment of this application, a device for determining violations is also provided, comprising: an acquisition module, configured to acquire image data collected by an image acquisition device for a target area within a preset time period, wherein the target area includes an area below a target device, and the image acquisition device is disposed at the bottom of the target device to acquire images of the target area; a obtaining module, configured to process multiple frames of images included in the image data to obtain alarm areas corresponding to the multiple frames of images and a movement trajectory of a first object within the target area; and a sending module, configured to determine whether the first object has committed a violation based on the last trajectory position of the movement trajectory and the positional distribution relationship of the alarm areas, wherein the violation indicates abnormal behavior associated with the target area.

[0020] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described method at runtime.

[0021] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described method through the computer program.

[0022] In this embodiment, after an image acquisition device located at the bottom of the target device acquires images of the area below the target device within a preset time period, the acquired image data is obtained. Then, the multiple frames of the image data are processed to obtain the alarm area corresponding to the multiple frames and the movement trajectory of the first object within the target area. Based on the final trajectory position and the positional distribution of the alarm area, it is determined whether the first object has engaged in any violations. This technical solution solves the technical problem that, when lifting and hoisting machinery moves along a track, the fixed camera area prevents movement monitoring of workers and accurate identification of their violations. It achieves movement monitoring of workers, thereby improving the accuracy of identifying violations. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a hardware structure block diagram of a computer terminal for a method of determining violations according to an embodiment of this application.

[0025] Figure 2 This is a flowchart of a method for determining violations according to an embodiment of this application;

[0026] Figure 3 This is a flowchart of a method for determining violations according to another embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the target area according to an optional embodiment of this application;

[0028] Figure 5(a) is a schematic diagram (a) of the alarm area according to an optional embodiment of this application;

[0029] Figure 5(b) is a schematic diagram (ii) of the alarm area according to an optional embodiment of this application;

[0030] Figure 6 This is a flowchart illustrating a method for determining violations according to one embodiment of this application;

[0031] Figure 7 This is a flowchart illustrating a method for determining violations according to yet another embodiment of this application;

[0032] Figure 8 This is a structural block diagram of a device for determining a violation according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of determining violations according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0036] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0038] This embodiment provides a method for determining violations, applied to a terminal device. Figure 2 This is a flowchart of a method for determining violations according to an embodiment of this application, the process including the following steps:

[0039] Step S202: Obtain image data collected by the image acquisition device for the target area within a preset time period, wherein the target area includes the area below the target device, and the image acquisition device is set at the bottom of the target device to acquire images of the target area;

[0040] It should be noted that the target device can be mounted on a guide rail and can move horizontally along the rail. During this horizontal movement, the image acquisition device mounted at the bottom of the target device also moves horizontally.

[0041] It should be noted that the aforementioned preset time period can be manually preset, and this application does not impose any restrictions on this. Furthermore, the image data collected within the aforementioned preset time period may include multiple frames of images from consecutive time periods.

[0042] Step S204: Process the multiple frames of images included in the image data to obtain the alarm area corresponding to the multiple frames of images and the movement trajectory of the first object in the target area;

[0043] Step S206: Based on the relationship between the last trajectory position of the movement trajectory and the location distribution of the alarm area, determine whether the first object has engaged in any illegal behavior, wherein the illegal behavior refers to abnormal behavior associated with the target area.

[0044] Through the above steps, image data collected by the image acquisition device within a preset time period is obtained for the target area. The target area includes the area below the target device, and the image acquisition device is positioned at the bottom of the target device to acquire images of the target area. The image data, comprising multiple frames, is processed to obtain the alarm area corresponding to each frame and the movement trajectory of the first object within the target area. Based on the final trajectory position and the positional distribution of the alarm area, it is determined whether the first object has engaged in any violations, where the violations represent abnormal behavior associated with the target area. This technical solution solves the technical problem that, when lifting and hoisting machinery moves along a track, the fixed camera area prevents movement monitoring of workers and accurate identification of their violations. It achieves movement monitoring of workers, thereby improving the accuracy of identifying violations.

[0045] It should be noted that the final trajectory position of the aforementioned action trajectory may include, but is not limited to, trajectory points corresponding to one or more frames. For example, if the final trajectory position of the aforementioned action trajectory includes trajectory points corresponding to one frame, the trajectory point corresponding to the current frame image can be determined as the final trajectory position of the aforementioned action trajectory; alternatively, the trajectory point corresponding to the last frame image in a multi-frame sequence can be determined as the final trajectory position of the aforementioned action trajectory. If the final trajectory position of the aforementioned action trajectory includes trajectory points corresponding to multiple frames, the final trajectory position of the action trajectory can be determined based on the trajectory points corresponding to the current frame image and other frames image preceding the current frame image; alternatively, the final trajectory position of the action trajectory can be determined based on the trajectory points corresponding to the last frame image in a multi-frame sequence and the images preceding the last frame image.

[0046] The present application does not impose any restrictions on the following: the current frame image and other frames before the current frame image may be defined as consecutive frames, or the last frame image and the images before the last frame image may be defined as consecutive frames.

[0047] There are many ways to implement step S204 above. In one exemplary embodiment, processing the image data including multiple frames of images to obtain alarm regions corresponding to the multiple frames of images includes: for each frame of the multiple frames of images, dividing each frame of images based on the geometric center of each frame of images to obtain divided multiple frames of images, each frame of the divided multiple frames of images having multiple division regions; sending a calibration prompt message containing the divided multiple frames of images to a second object, and receiving feedback information sent by the second object in response to the calibration prompt message, wherein the feedback information includes the result of the second object calibrating the region type of the multiple division regions of each frame of the divided multiple frames of images, wherein the region type includes: alarm region and non-alarm region; determining the division region indicated in the feedback information as an alarm region as the alarm region corresponding to the multiple frames of images.

[0048] In other words, through the above embodiments, multiple frames of images can be divided to obtain multiple region types of each frame of image. Based on the region type of the division area which is an alarm region, the alarm region corresponding to multiple frames of images can be determined. This enables the dynamic determination of alarm regions of multiple frames of images, laying the foundation for the subsequent process of determining whether the movement trajectory of the first object in the target area is located in the alarm region. This provides the preparation conditions for sending alarm information and improves the executability of sending alarm information.

[0049] In an exemplary embodiment, step S204 above can also be implemented by the following steps: acquiring the current frame image and other frame images preceding the current frame image in the multi-frame images, and determining the mapping relationship between the current frame image and the other frame images; wherein, the mapping relationship represents the mathematical relationship required to convert the coordinate position of the first object in the other frame images into the coordinate position in the coordinate system of the current frame image; acquiring the first coordinate position of the first object in the other frame images, and determining the second coordinate position corresponding to the first coordinate position in the coordinate system of the current frame image according to the mapping relationship; acquiring the third coordinate position of the first object in the coordinate system of the current frame image; and determining the movement trajectory according to the plurality of second coordinate positions and the third coordinate position.

[0050] Furthermore, in one embodiment, the process of determining the movement trajectory based on multiple second coordinate positions and the third coordinate position is described as follows: The sum of the following parameters is determined: the third coordinate position, the position difference between the second coordinate position and the third coordinate position; a fourth coordinate position in the coordinate system of the current frame image is determined based on the sum, wherein the fourth coordinate position is the position of the fifth object in the coordinate system of the current frame image; a first object feature is determined when the fifth object is located at the fourth coordinate position, a second object feature corresponding to the first object being located at the second coordinate position, and a third object feature corresponding to the first object being located at the third coordinate position; if the feature overlap between the first object feature and the second object feature is greater than a preset value, and the feature overlap between the first object feature and the third object feature is greater than the preset value, the fifth object is determined to be consistent with the first object, and the movement trajectory of the first object in the current frame image is determined based on the second coordinate position, the third coordinate position, and the fourth coordinate position.

[0051] In one embodiment, the position difference between a plurality of second coordinate positions and the third coordinate position can be determined by: determining the second coordinate position corresponding to the first frame image of a plurality of other frame images from the plurality of second coordinate positions, the second coordinate position corresponding to the last frame image of a plurality of other frame images, and determining the position offset value between the second coordinate position corresponding to the first frame image of a plurality of other frame images and the second coordinate position corresponding to the last frame image of a plurality of other frame images; obtaining the number of frames of a plurality of other frame images, determining the ratio of the position offset to the number of frames as the target second coordinate position, and determining the position difference between the target second coordinate position and the third coordinate position as the position difference between the plurality of second coordinate positions and the third coordinate position.

[0052] In this embodiment, for example, the first frame of the multiple other frame images can be taken as the first frame image, and the last frame of the multiple other frame images can be taken as the tenth frame image. Then, the number of frames of the multiple other frame images is 10, and the ratio of the position offset to 10 is determined as the position difference between the multiple second coordinate positions and the third coordinate positions.

[0053] Optionally, in other embodiments, the movement trajectory can also be determined based on a single second coordinate position and the third coordinate position. Specifically, the following parameters are determined: the third coordinate position, the position difference between the second coordinate position and the third coordinate position; a fourth coordinate position in the coordinate system of the current frame image is determined based on the sum; and the movement trajectory of the first object in the current frame image is determined based on the second coordinate position, the third coordinate position, and the fourth coordinate position.

[0054] Furthermore, determining the movement trajectory of the first object in the current frame image based on the second coordinate position, the third coordinate position, and the fourth coordinate position can also be achieved through the following steps: If the feature overlap between the first object feature and the second object feature is greater than a preset value, and the feature overlap between the first object feature and the third object feature is greater than the preset value, then the fifth object is determined to be consistent with the first object. The movement trajectory of the first object in the current frame image is then determined based on the second coordinate position, the third coordinate position, and the fourth coordinate position. The first object feature is the object feature of the fifth object when it is located at the fourth coordinate position; the second object feature is the object feature corresponding to the first object when it is located at the second coordinate position; and the third object feature is the object feature corresponding to the first object when it is located at the third coordinate position.

[0055] Compared to the technical solution of determining the movement trajectory based on a single second coordinate position and the third coordinate position, determining the movement trajectory based on multiple second coordinate positions and the third coordinate position can reduce the position error that occurs during the coordinate position transformation process, obtain a more accurate coordinate position, and thus improve the accuracy of determining the movement trajectory.

[0056] Through the above embodiments, the first object's movement trajectory within the target area is determined by the mapping relationship between the current frame image and the other frame images, using the second coordinate position of the first object in the coordinate system of the current frame image and the third coordinate position of the first object in the coordinate system of the current frame image. This allows the movement trajectory to dynamically represent the positional changes of the first object in multiple frames, achieving the technical objective of dynamically monitoring the first object. This reduces the probability of failing to promptly alert the first object due to its inability to be detected, thereby improving the security of the first object.

[0057] Optionally, in other embodiments, if it is determined that the feature overlap between the first object feature and the second object feature is less than a preset value, or the feature overlap between the first object feature and the third object feature is less than the preset value, then it is determined that the fifth object is inconsistent with the first object, and the movement trajectory of the fifth object in the current frame image is determined based on the fourth coordinate position.

[0058] Through the above embodiments, even when the fifth object is inconsistent with the first object, the movement trajectory of the fifth object in the current frame image can be determined, thereby enabling independent monitoring of the first object and the fifth object, and further realizing the dynamic monitoring process of multiple targets.

[0059] In an exemplary embodiment, a technical solution for determining the mapping relationship between the current frame image and the other frame images is further proposed. The specific steps include: when it is determined that a third object exists in both the current frame image and the other frame images, obtaining a first image feature of the third object in the current frame image and a second image feature of the third object in the other frame images; calculating the transformation matrix between the first image feature and the second image feature according to a preset algorithm to obtain the mapping relationship between the first image feature and the second image feature.

[0060] It should be noted that the above-mentioned preset algorithm can be understood as the matching algorithm when performing feature matching, including but not limited to the Hamming distance matching algorithm, the FLANN fast nearest neighbor matching algorithm, and the local matching algorithm in optical flow calculation.

[0061] The first image feature and the second image feature mentioned above can be point features of the image, and the point feature extraction methods include, but are not limited to, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented Fast and Rotated BRIEF), optical flow, etc.

[0062] Through the above embodiments, by providing a detailed explanation of the mapping relationship between the first image features and the second image features, the reliability of calculating the movement trajectory of the first object within the target area can be improved, thereby improving the accuracy of determining whether the final trajectory position of the movement trajectory is located within the alarm area.

[0063] In an exemplary embodiment, before performing the step S204 above, which involves processing the multiple frames of images included in the image data to obtain the movement trajectory of the first object within the target area, the first object can be determined from the multiple frames of images. Specifically, the process of determining the first object from the multiple frames of images includes: acquiring different objects identified in the multiple frames of images; identifying the different objects based on their object characteristics to obtain the object types corresponding to the different objects; acquiring a fourth object with the same object type among the different objects; and determining the fourth object as the first object if the identity of the fourth object matches the identity type of the first object.

[0064] Through the above embodiments, the types of different objects in the multi-frame images can be obtained, and the first object can be determined according to the object type corresponding to the different objects, which greatly improves the accuracy of determining the first object.

[0065] In an exemplary embodiment, further, in the process of acquiring different objects identified in the multi-frame images, a first object area preset for each of the different objects can be determined, and a second object area of ​​each object in the multi-frame images can be acquired; if the difference between the first object area and the second object area is less than or equal to a preset value, the object feature corresponding to the first object area is determined as the object feature of each object.

[0066] Through the above embodiments, the object characteristics of different objects can be determined by the object area of ​​different objects. For example, if the different objects include a sixth object, and the above preset value is 1900, then if the first object area of ​​the sixth object is 100×100 and the second object area of ​​the sixth object is 90×90, the difference between the first object area and the second object area is 1900, then the object characteristics corresponding to the first object area can be determined as the object characteristics of the sixth object.

[0067] Furthermore, object features can be determined by combining the aspect ratios of different objects. For example, if the area of ​​the first object of the sixth object is 100×100 and the aspect ratio of the sixth object is 0.5:1.5, the object features corresponding to the area of ​​the first object can be determined as the object features of the sixth object.

[0068] Optionally, in other embodiments, the object type of the first object can be determined directly by the object area of ​​different objects. For example, if the difference between the area of ​​the first object and the area of ​​the second object is less than or equal to a preset value, and if it is determined that the area of ​​the first object corresponds to an object type, then the object type corresponding to the area of ​​the first object can be directly determined as the object type of the first object.

[0069] In an exemplary embodiment, to better understand the process of determining whether the first object has violated regulations based on the location relationship between the last trajectory position of the action trajectory and the location distribution of the alarm area in step S206 above, the following technical solution can be proposed: if the last trajectory position is determined to be within the alarm area, the first object is determined to have violated regulations; if the last trajectory position is determined not to be within the alarm area, the first object is determined not to have violated regulations.

[0070] Furthermore, the process of determining that the final trajectory position of the action trajectory is located within the alarm area is described according to the following technical solution, specifically including: obtaining a preset number of continuous trajectory points from all trajectory points included in the final trajectory position; obtaining each trajectory ray of each continuous trajectory point at a preset angle; if it is determined that each trajectory ray intersects with the boundary line of the alarm area, obtaining the number of intersections between each trajectory ray and the boundary line of the alarm area; if it is determined that the number of intersections between each trajectory ray and the boundary line of the alarm area satisfies a preset condition, and if the final trajectory position of the action trajectory is located within the alarm area, then it is determined that the first object has committed a violation.

[0071] In an exemplary embodiment, after determining that the first object has committed a violation, the method further includes: sending an alarm message to the first object if the last trajectory position is located within the alarm area.

[0072] Through the above embodiments, it is possible to determine that the final trajectory position of the movement trajectory is located within the alarm area, thereby achieving the technical objective of sending alarm information to the first object and improving the security of the first object.

[0073] In an exemplary embodiment, the following technical solution is proposed to describe the process of determining that the final trajectory position of the action trajectory is not located within the alarm area, specifically including: obtaining a preset number of continuous trajectory points from all trajectory points included in the final trajectory position; obtaining any trajectory ray of any trajectory point of the continuous trajectory points at the preset angle; if it is determined that the number of intersections between the any trajectory ray and the boundary line of the alarm area does not meet the preset condition, it is determined that the final trajectory position of the action trajectory is not located within the alarm area, and thus it is determined that the first object does not have any violation behavior.

[0074] Optionally, after determining that the final trajectory position of the movement trajectory is not within the alarm area, the coordinate position corresponding to any trajectory point can be stored, and a preset number of other continuous trajectory points can be obtained from all trajectory points included in the final trajectory position. Based on the other continuous trajectory points, it can be determined whether the final trajectory position of the movement trajectory is within the alarm area.

[0075] The above embodiments provide a solution for determining that the final trajectory position of the action trajectory is not within the alarm area. By acquiring a preset number of consecutive trajectory points multiple times, it can be determined that the final trajectory position of the action trajectory is not within the alarm area, thereby improving the accuracy of sending alarm information to the first object when the final trajectory position of the action trajectory is determined to be within the alarm area.

[0076] In the above embodiments, the preset condition can be, for example, an odd number. If the number of intersections between each trajectory ray and the boundary line of the alarm area satisfies the preset condition (i.e., the number of intersections is odd), then the final trajectory position of the movement trajectory is determined to be within the alarm area. Conversely, if the number of intersections between any trajectory ray and the boundary line of the alarm area does not satisfy the preset condition (i.e., the number is not odd, for example, if it is even), then the final trajectory position of the movement trajectory is determined not to be within the alarm area.

[0077] This embodiment provides a method for determining violations, applied to a server. Figure 3 This is a flowchart of a method for determining violations according to another embodiment of this application, the process including the following steps:

[0078] Step S302: The mobile crane is installed on the guide rail (i.e., the track) and moves horizontally. The camera is directly installed on the equipment and shoots downwards. Initially, the perimeter alarm area (i.e. the alarm area mentioned above) is determined by manually marking lines or by an automated line. As the camera moves with the equipment, the relative position of the perimeter alarm area in the video image remains unchanged.

[0079] Step S302 can also be implemented in the following ways: Figure 4 As shown, Figure 4 This is a schematic diagram of the target area according to an optional embodiment of this application. The camera (i.e., the above-mentioned image acquisition device) can be directly installed on a mechanical device such as a mobile crane to take pictures downwards. The device 2 is installed on the guide rail 1 and can move freely on the guide rail 1 along the horizontal moving direction. The camera 3 is installed on the device 2 and the shooting direction is vertically downwards, aimed at the ground 4.

[0080] Furthermore, after acquiring the video surveillance images captured by the camera, the perimeter alarm area of ​​the video surveillance image is determined by manual or automatic line drawing. Manual line drawing includes, but is not limited to, user-defined perimeter alarm areas based on scene characteristics and actual needs; automatic line drawing includes, but is not limited to, image segmentation and classification followed by user-defined or preset perimeter alarm areas. The perimeter alarm area can be set at the center of the image, indicating the danger zone below the equipment. If any personnel are within this danger zone, an alarm must be triggered to alert them.

[0081] As the camera moves continuously with the crane, the video footage captured by the camera is constantly changing, and correspondingly, different video footage has different perimeter alarm areas. In one embodiment, the perimeter alarm area can be described in conjunction with Figures 5(a) and 5(b). As shown in Figures 5(a) and 5(b), the area bounded by the dashed perimeter lines is the perimeter alarm area. In Figures 5(a) and 5(b), the relative position of the perimeter alarm area in the video footage (i.e., the position and area of ​​the perimeter area relative to the camera's field of view) remains unchanged, but the actual corresponding world / physical coordinates are constantly changing, and there is also movement of people below.

[0082] Step S304: Acquire video images within a certain time interval during the camera's movement, perform target detection on the images respectively, and obtain the position and bounding box information of each target in the image;

[0083] In other words, after acquiring multiple frames of video images within a certain time interval during the movement of the camera with the device, these multiple frames of video images can be recorded as frames k to k+n-1 (n can be set freely). Target detection is then performed on each image to obtain the corresponding position and feature information of each target in the image, where k and n are both natural numbers.

[0084] The target detection methods include, but are not limited to, moving target detection methods and deep learning-based target detection networks. The results of target detection can be filtered based on target category, target size, etc. For example, after filtering, only the target categories of interest can be retained, and the location information corresponding to each target can be obtained based on the target categories of interest. Simultaneously, target feature information (which can be used for subsequent target tracking) and image information can be saved, thereby maintaining a data queue of length n containing target feature information.

[0085] Among them, the video images within a certain time interval can be shown in Figure 5(a) and Figure 5(b). Figure 5(a) and Figure 5(b) are consecutive frame images. Figure 5(a) represents the image corresponding to "2022-04-13 13:52:00", while Figure 5(b) represents "2022-04-13 13:52:01". In Figure 5(a), person A is crossing the track, and in Figure 5(b), person A has already crossed the track.

[0086] Step S306: Obtain the current frame image and the mapping relationship between other frame images before the current frame. Based on the mapping relationship between different images, obtain the target position of each target (i.e. the first object mentioned above) in the current frame image through coordinate transformation.

[0087] Step S306 above includes the following implementation steps:

[0088] Step 1: Obtain the current frame image, denoted as frame k+n.

[0089] Step 2: Obtain the mapping relationship between the current frame image and the images from frame k to frame (k+n-1).

[0090] Step 3: Based on the mapping relationship between different images, obtain the corresponding target positions of each target in the previous frame in the current frame through coordinate transformation.

[0091] Specifically, a matching algorithm is used to obtain the mapping relationship between the k-th to k+n-1-th frames and the k+n-th frame. The matching algorithm includes, but is not limited to, grayscale-based matching algorithms and feature-based matching algorithms.

[0092] Taking feature-based matching algorithms as an example, the first step is to extract image features, such as point features. Point feature extraction methods include, but are not limited to, SIFT, SURF, ORB, and optical flow. After obtaining the point features, the target detection results can be used for filtering. During the filtering process, in order to reduce the external influence of moving targets, the point features can be set to the features of static objects, or feature points within the target bounding box and related feature points near the point features can be deleted. Secondly, point matching algorithms are used to match feature points. Point matching algorithms include, but are not limited to, Hamming distance-based matching, FLANN fast nearest neighbor matching, and local matching in optical flow calculation. Finally, the RANSAC algorithm is used to filter out incorrect matching points, and the transformation matrix between the target position in the (k+n-1)th frame and the target position in the (k+n)th frame is calculated. The resulting homography matrix is ​​the corresponding mapping relationship between the two images.

[0093] By combining the mapping relationship between the kth to k+n-1th frames and the k+nth frame, the target position and target size of each target in the kth to k+n-1th frames can be obtained in the current k+nth frame through coordinate transformation.

[0094] Optionally, the above embodiments can be achieved through... Figure 6 The flowchart is used for illustration. Figure 6 This is a flowchart illustrating a method for determining violations according to an embodiment of this application, as shown below. Figure 6 As shown, the specific steps are as follows:

[0095] Step S602: Extract point features from the two images;

[0096] Step S604: Based on the target detection results and the type of target of interest, filter the target detection results and remove relevant feature points within the target bounding box and near the point features;

[0097] Step S606: Use a point matching algorithm to match feature points;

[0098] Step S608: Use the RANSAC algorithm to filter out erroneous matching points and obtain the homography matrix;

[0099] Step S610: Combining the mapping relationship between the current frame image and other frame images, obtain the target positions in the current frame image corresponding to each target in the previous frame images through coordinate transformation.

[0100] Step S308: Target detection is performed on the current frame image. The position information (i.e., the coordinate position) and size information (equivalent to the object area) of the target in other frames before the current frame are input together to perform data association, update the target's motion trajectory, and multi-target tracking can also be performed on multiple targets in the current frame image.

[0101] It should be noted that data association methods include, but are not limited to, SORT and DEEPSORT. Taking DEEPSORT as an example, for a single target in the frame, the target's coordinate-transformed position and its position in the current frame are used as input for trajectory association and prediction. The position difference between the target's coordinate-transformed position and its position in the current frame is used to predict the next target's position, resulting in the predicted trajectory. IOU matching and feature matching are performed on the coordinate-transformed target bounding boxes. When a match is successful, the same target in different frames can be effectively associated and matched, effectively reducing the problems of the same target not being associated between different frames due to video frame movement and misassociation between different targets. When a match fails, a new target is obtained, and association and matching are performed on the new target in different frames. Furthermore, by associating and matching each target in different frames, accurate multi-target tracking can be achieved.

[0102] The target position after coordinate transformation can be the target position obtained by transforming the target in a single frame image or the target position obtained by transforming the target in multiple frames images. This application does not limit this.

[0103] Step S310: Determine whether the target has triggered a perimeter alarm based on the perimeter alarm area and the target's movement trajectory. If the perimeter alarm is triggered, that is, the last trajectory position of the movement trajectory is located within the alarm area, then send alarm information to the first object.

[0104] In step S310, the process of determining whether the target has triggered a perimeter alarm based on the perimeter alarm area and the target's movement trajectory is as follows: the process of determining whether each trajectory point in the movement trajectory is within the perimeter alarm area is determined by a discrimination method, wherein the discrimination method includes, but is not limited to, area and discrimination method, angle and discrimination method, light projection method, etc.

[0105] Taking the ray projection method as an example, a ray is drawn from the point of motion trajectory. The number of intersections with the boundary line of the perimeter alarm area is calculated. If the number of intersections is odd, the point is inside the area; if the number of intersections is even, the point is outside the area. In other words, for example, if the number of intersections with the boundary line is 1, it can be understood that the ray is tangent to the boundary line, hence there is only one intersection. Or, if the number of intersections with the boundary line is 2, it can be understood that the ray enters the perimeter alarm area and then exits, resulting in 2 intersections.

[0106] Subsequently, the status can be determined based on whether each trajectory point is within the perimeter alarm area. If the trajectory point corresponding to the image frame (such as the current frame and the three most recent frames) is within the alarm area within a certain time interval, an alarm is triggered and the target location is output. If it is outside the alarm area, the corresponding target location information, feature information and image information are saved, and the loop of the next frame continues.

[0107] Through the above embodiments, in the scenario of lifting and hoisting, a camera is directly mounted on the moving crane, facing downwards to capture images. The camera moves continuously with the crane, and during this movement, the relative position of the perimeter alarm area in the video frame remains unchanged. Target detection is performed on the video images acquired during camera movement to obtain the corresponding positions of each target within the image, and the positional mapping relationship between different images is obtained. Based on the positional mapping relationship between different images, coordinate transformation is performed to obtain the position of the target in other frames before the current frame in the current frame image. Data association is then performed by combining the position of the target in other frames with the position in the current frame image to update the target's motion trajectory. Thus, based on the perimeter alarm area and the target's motion trajectory, it is determined whether the target has triggered the perimeter alarm. This allows for alarms to be triggered when personnel illegally appearing under the crane during its movement, reducing the occurrence of dangerous incidents.

[0108] To better understand the process of the above method, the implementation method flow will be further described below with reference to optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0109] In another exemplary embodiment, combined with Figure 7 The process of determining whether each trajectory point of the motion trajectory exists within the alarm area is explained. Figure 7 This is a flowchart illustrating a method for determining violations according to yet another embodiment of this application, as shown below. Figure 7 As shown, the specific steps are as follows:

[0110] Step S702: Determine whether each trajectory point of the motion trajectory is within the perimeter alarm area;

[0111] Step S704: Further determine whether the trajectory points corresponding to the image within a certain time interval are within the perimeter alarm area; if yes, proceed to step S706; if no, proceed to step S70.

[0112] Step S706: Issue an alarm and output the target location.

[0113] Step S708: Update the motion trajectory and continue processing the next frame image.

[0114] The above embodiments enable perimeter alarm technology, which not only prevents unauthorized entry and exit within a designated area but also plays a crucial role in dangerous locations such as railways, highways, and airports, as well as important locations like government offices and protected areas. Furthermore, by introducing computer vision methods into intelligent monitoring and establishing a mapping relationship between images and image descriptions, computers can analyze and understand the content of video footage. By customizing alarm zones or tripwires within the video feed and combining them with target recognition, detection, and tracking technologies, an alarm is automatically triggered when a target enters the zone or crosses the tripwire. Compared to personnel supervision, this method is not only safer and more effective but also requires significantly less manpower.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0116] Figure 8 This is a structural block diagram of a device for determining a violation according to an embodiment of this application; as shown below. Figure 8 As shown, it includes:

[0117] The acquisition module 802 is used to acquire image data collected by the image acquisition device for a target area within a preset time period, wherein the target area includes the area below the target device, and the image acquisition device is set at the bottom of the target device to acquire images of the target area;

[0118] It should be noted that the target device can be mounted on a guide rail and can move horizontally along the rail. During this horizontal movement, the image acquisition device mounted at the bottom of the target device also moves horizontally.

[0119] It should be noted that the aforementioned preset time period can be manually preset, and this application does not impose any restrictions on this. Furthermore, the image data collected within the aforementioned preset time period may include multiple frames of images from consecutive time periods.

[0120] The module 804 is used to process the multiple frames of images included in the image data to obtain the alarm area corresponding to the multiple frames of images and the movement trajectory of the first object in the target area;

[0121] The sending module 806 is used to determine whether the first object has violated regulations based on the last trajectory position of the movement trajectory and the location distribution relationship of the alarm area, wherein the violation refers to abnormal behavior associated with the target area.

[0122] The above-described device acquires image data collected by an image acquisition device within a preset time period for a target area, wherein the target area includes the area below the target device, and the image acquisition device is positioned at the bottom of the target device to acquire images of the target area. The device processes multiple frames of images to obtain alarm areas corresponding to the multiple frames and the movement trajectory of a first object within the target area. Based on the final trajectory position and the positional distribution relationship of the alarm areas, it determines whether the first object has engaged in any violations, where the violations represent abnormal behavior associated with the target area. This technical solution solves the technical problem that, when lifting and hoisting machinery moves along a track, the fixed camera area prevents movement monitoring of workers and accurate identification of their violations. It achieves movement monitoring of workers, thereby improving the accuracy of identifying violations.

[0123] It should be noted that the final trajectory position of the aforementioned action trajectory may include, but is not limited to, trajectory points corresponding to one or more frames. For example, if the final trajectory position of the aforementioned action trajectory includes trajectory points corresponding to one frame, the trajectory point corresponding to the current frame image can be determined as the final trajectory position of the aforementioned action trajectory; alternatively, the trajectory point corresponding to the last frame image in a multi-frame sequence can be determined as the final trajectory position of the aforementioned action trajectory. If the final trajectory position of the aforementioned action trajectory includes trajectory points corresponding to multiple frames, the final trajectory position of the action trajectory can be determined based on the trajectory points corresponding to the current frame image and other frames image preceding the current frame image; alternatively, the final trajectory position of the action trajectory can be determined based on the trajectory points corresponding to the last frame image in a multi-frame sequence and the images preceding the last frame image.

[0124] The present application does not impose any restrictions on the following: the current frame image and other frames before the current frame image may be defined as consecutive frames, or the last frame image and the images before the last frame image may be defined as consecutive frames.

[0125] In an exemplary embodiment, the above-described obtaining module 804 is further configured to: for each frame of the multi-frame image, divide each frame of the multi-frame image based on the geometric center of each frame of the multi-frame image to obtain a multi-frame image after division, wherein each frame of the multi-frame image after division has multiple division regions; send a calibration prompt message containing the multi-frame image after division to a second object, and receive feedback information sent by the second object in response to the calibration prompt message, wherein the feedback information includes the result of the second object calibrating the region type of the multiple division regions of each frame of the multi-frame image after division, wherein the region type includes: alarm region and non-alarm region; determine the division region whose region type is alarm region indicated in the feedback information as the alarm region corresponding to the multi-frame image.

[0126] In other words, through the above embodiments, multiple frames of images can be divided to obtain multiple region types of each frame of image. Based on the region type of the division area which is an alarm region, the alarm region corresponding to multiple frames of images can be determined. This enables the dynamic determination of alarm regions of multiple frames of images, laying the foundation for the subsequent process of determining whether the movement trajectory of the first object in the target area is located in the alarm region. This provides the preparation conditions for sending alarm information and improves the executability of sending alarm information.

[0127] In an exemplary embodiment, the above-described obtaining module 804 is further configured to: obtain the current frame image and other frame images preceding the current frame image in the multi-frame images, and determine the mapping relationship between the current frame image and the other frame images; wherein the mapping relationship represents the mathematical relationship required to convert the coordinate position of the first object in the other frame images into the coordinate position in the coordinate system of the current frame image; obtain the first coordinate position of the first object in the other frame images, and determine the second coordinate position corresponding to the first coordinate position in the coordinate system of the current frame image according to the mapping relationship; obtain the third coordinate position of the first object in the coordinate system of the current frame image; and determine the movement trajectory according to the plurality of second coordinate positions and the third coordinate position.

[0128] Furthermore, the module 804 described above is also used to: determine the sum of the following parameters: the third coordinate position, the position difference between the second coordinate position and the third coordinate position; determine the fourth coordinate position in the coordinate system of the current frame image based on the sum, wherein the fourth coordinate position is the position of the fifth object in the coordinate system of the current frame image; determine the first object feature when the fifth object is located at the fourth coordinate position, the second object feature corresponding to the first object when it is located at the second coordinate position, and the third object feature corresponding to the first object when it is located at the third coordinate position; if the feature overlap between the first object feature and the second object feature is greater than a preset value, and the feature overlap between the first object feature and the third object feature is greater than the preset value, determine that the fifth object is consistent with the first object, and determine the movement trajectory of the first object in the current frame image based on the second coordinate position, the third coordinate position, and the fourth coordinate position.

[0129] The module 804 described above is further configured to: determine the position difference between a plurality of second coordinate positions and the third coordinate position by: determining the second coordinate position corresponding to the first frame image of a plurality of other frame images from the plurality of second coordinate positions, the second coordinate position corresponding to the last frame image of the other frame images, and determining the position offset value between the second coordinate position corresponding to the first frame image of the other frame images and the second coordinate position corresponding to the last frame image of the other frame images; obtaining the number of frames of the plurality of other frame images, determining the ratio of the position offset to the number of frames as the target second coordinate position, and determining the position difference between the target second coordinate position and the third coordinate position as the position difference between the plurality of second coordinate positions and the third coordinate position.

[0130] Through the above embodiments, the first object's movement trajectory within the target area is determined by the mapping relationship between the current frame image and the other frame images, using the second coordinate position of the first object in the coordinate system of the current frame image and the third coordinate position of the first object in the coordinate system of the current frame image. This allows the movement trajectory to dynamically represent the positional changes of the first object in multiple frames, achieving the technical objective of dynamically monitoring the first object. This reduces the probability of failing to promptly alert the first object due to its inability to be detected, thereby improving the security of the first object.

[0131] Optionally, in other embodiments, if it is determined that the feature overlap between the first object feature and the second object feature is less than a preset value, or the feature overlap between the first object feature and the third object feature is less than the preset value, then it is determined that the fifth object is inconsistent with the first object, and the movement trajectory of the fifth object in the current frame image is determined based on the fourth coordinate position.

[0132] Through the above embodiments, even when the fifth object is inconsistent with the first object, the movement trajectory of the fifth object in the current frame image can be determined, thereby enabling independent monitoring of the first object and the fifth object, and further realizing the dynamic monitoring process of multiple targets.

[0133] In an exemplary embodiment, the above-described obtaining module 804 is further configured to: when it is determined that a third object exists in both the current frame image and the other frame images, obtain a first image feature of the third object in the current frame image and a second image feature of the third object in the other frame images; calculate a transformation matrix between the first image feature and the second image feature according to a preset algorithm to obtain a mapping relationship between the first image feature and the second image feature.

[0134] It should be noted that the above-mentioned preset algorithm can be understood as the matching algorithm when performing feature matching, including but not limited to the Hamming distance matching algorithm, the FLANN fast nearest neighbor matching algorithm, and the local matching algorithm in optical flow calculation.

[0135] The first image feature and the second image feature mentioned above can be point features of the image, and the point feature extraction methods include, but are not limited to, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented Fast and Rotated BRIEF), optical flow, etc.

[0136] Through the above embodiments, by providing a detailed explanation of the mapping relationship between the first image features and the second image features, the reliability of calculating the movement trajectory of the first object within the target area can be improved, thereby improving the accuracy of determining whether the final trajectory position of the movement trajectory is located within the alarm area.

[0137] In an exemplary embodiment, the above-described obtaining module 804 is further configured to: determine the first object from the multi-frame images, wherein determining the first object from the multi-frame images includes: acquiring different objects identified in the multi-frame images; identifying the different objects based on their object features to obtain object types corresponding to the different objects; acquiring a fourth object among the different objects that has the same object type, and determining the fourth object as the first object if the identity of the fourth object is consistent with the identity type of the first object.

[0138] Through the above embodiments, the types of different objects in the multi-frame images can be obtained, and the first object can be determined according to the object type corresponding to the different objects, which greatly improves the accuracy of determining the first object.

[0139] In an exemplary embodiment, the above-described obtaining module 804 is further configured to: for each of the different objects, determine a first object area preset for each object, and obtain a second object area of ​​each object in the multi-frame image; if the difference between the first object area and the second object area is less than or equal to a preset value, determine the object feature corresponding to the first object area as the object feature of each object.

[0140] Through the above embodiments, the object characteristics of different objects can be determined by the object area of ​​different objects. For example, if the different objects include a sixth object, and the above preset value is 1900, then if the first object area of ​​the sixth object is 100×100 and the second object area of ​​the sixth object is 90×90, the difference between the first object area and the second object area is 1900, then the object characteristics corresponding to the first object area can be determined as the object characteristics of the sixth object.

[0141] Furthermore, object features can be determined by combining the aspect ratios of different objects. For example, if the area of ​​the first object of the sixth object is 100×100 and the aspect ratio of the sixth object is 0.5:1.5, the object features corresponding to the area of ​​the first object can be determined as the object features of the sixth object.

[0142] Optionally, in other embodiments, the object type of the first object can be determined directly by the object area of ​​different objects. For example, if the difference between the area of ​​the first object and the area of ​​the second object is less than or equal to a preset value, and if it is determined that the area of ​​the first object corresponds to an object type, then the object type corresponding to the area of ​​the first object can be directly determined as the object type of the first object.

[0143] In an exemplary embodiment, the sending module 806 is further configured to: obtain a preset number of continuous trajectory points from all trajectory points included in the last trajectory position; obtain each trajectory ray of each continuous trajectory point at a preset angle; if it is determined that each trajectory ray intersects with the boundary line of the alarm area, obtain the number of intersections between each trajectory ray and the boundary line of the alarm area; if it is determined that the number of intersections between each trajectory ray and the boundary line of the alarm area satisfies a preset condition, and if the last trajectory position of the action trajectory is located within the alarm area, then determine that the first object has committed a violation.

[0144] Furthermore, the aforementioned sending module 806 is also used to: after determining that the first object has committed a violation, the method further includes: if the last trajectory position is located within the alarm area, sending alarm information to the first object.

[0145] Through the above embodiments, it is possible to determine that the final trajectory position of the movement trajectory is located within the alarm area, thereby achieving the technical objective of sending alarm information to the first object and improving the security of the first object.

[0146] In an exemplary embodiment, the sending module 806 is further configured to: obtain a preset number of continuous trajectory points from all trajectory points included in the last trajectory position, obtain any trajectory ray of any trajectory point of the continuous trajectory points at the preset angle; if the number of intersections between any trajectory ray and the boundary line of the alarm area does not meet the preset condition, determine that the last trajectory position of the action trajectory is not located within the alarm area, and then determine that the first object does not have any violation behavior.

[0147] Optionally, after determining that the final trajectory position of the movement trajectory is not within the alarm area, the coordinate position corresponding to any trajectory point can be stored, and a preset number of other continuous trajectory points can be obtained from all trajectory points included in the final trajectory position. Based on the other continuous trajectory points, it can be determined whether the final trajectory position of the movement trajectory is within the alarm area.

[0148] The above embodiments provide a solution for determining that the final trajectory position of the action trajectory is not within the alarm area. By acquiring a preset number of consecutive trajectory points multiple times, it can be determined that the final trajectory position of the action trajectory is not within the alarm area, thereby improving the accuracy of sending alarm information to the first object when the final trajectory position of the action trajectory is determined to be within the alarm area.

[0149] In the above embodiments, the preset condition can be, for example, an odd number. If the number of intersections between each trajectory ray and the boundary line of the alarm area satisfies the preset condition (i.e., the number of intersections is odd), then the final trajectory position of the movement trajectory is determined to be within the alarm area. Conversely, if the number of intersections between any trajectory ray and the boundary line of the alarm area does not satisfy the preset condition (i.e., the number is not odd, for example, if it is even), then the final trajectory position of the movement trajectory is determined not to be within the alarm area.

[0150] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0151] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0152] S11, acquire image data collected by the image acquisition device for the target area within a preset time period, wherein the target area includes the area below the target device, and the image acquisition device is set at the bottom of the target device to acquire images of the target area;

[0153] S12, process the multiple frames of images included in the image data to obtain the alarm area corresponding to the multiple frames of images and the movement trajectory of the first object in the target area;

[0154] S13, based on the last trajectory position of the movement trajectory and the location distribution relationship of the alarm area, determine whether the first object has violated regulations, wherein the violation refers to abnormal behavior associated with the target area.

[0155] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0156] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0157] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0158] S11, acquire image data collected by the image acquisition device for the target area within a preset time period, wherein the target area includes the area below the target device, and the image acquisition device is set at the bottom of the target device to acquire images of the target area;

[0159] S12, process the multiple frames of images included in the image data to obtain the alarm area corresponding to the multiple frames of images and the movement trajectory of the first object in the target area;

[0160] S13, based on the last trajectory position of the movement trajectory and the location distribution relationship of the alarm area, determine whether the first object has violated regulations, wherein the violation refers to abnormal behavior associated with the target area.

[0161] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0162] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0163] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0164] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining a violation, characterized in that, include: Image data acquired by an image acquisition device within a preset time period for a target area, wherein the target area includes the area below the target device, and the image acquisition device is disposed at the bottom of the target device to acquire images of the target area; The image data, including multiple frames, is processed to obtain the alarm area corresponding to the multiple frames and the movement trajectory of the first object within the target area; Based on the relationship between the last trajectory position of the movement trajectory and the location distribution of the alarm area, it is determined whether the first object has engaged in any illegal behavior, wherein the illegal behavior refers to abnormal behavior associated with the target area; The method further includes: the final trajectory position of the action trajectory includes trajectory points corresponding to one or more frames; when the final trajectory position of the action trajectory includes trajectory points corresponding to one frame, the trajectory point corresponding to the current frame image is determined as the final trajectory position of the action trajectory, or the trajectory point corresponding to the last frame image in the multi-frame images is determined as the final trajectory position of the action trajectory; when the final trajectory position of the action trajectory includes trajectory points corresponding to multiple frames, the final trajectory position of the action trajectory is determined based on the trajectory points corresponding to the current frame image and other frame images before the current frame image, or the final trajectory position of the action trajectory is determined based on the trajectory points corresponding to the last frame image in the multi-frame images and the images before the last frame image. The step of processing the image data, which includes multiple frames, to obtain the alarm area corresponding to the multiple frames and the movement trajectory of the first object within the target area includes: acquiring the current frame image and other frames preceding the current frame image from the multiple frames, and determining the mapping relationship between the current frame image and the other frames; acquiring the first coordinate position of the first object in the other frames, and determining the second coordinate position corresponding to the first coordinate position in the coordinate system of the current frame image based on the mapping relationship; acquiring the third coordinate position of the first object in the coordinate system of the current frame image; determining the sum of the following parameters: the third coordinate position, the position difference between the second coordinate position and the third coordinate position; determining the fourth coordinate position in the coordinate system of the current frame image based on the sum; and determining the movement trajectory of the first object in the current frame image based on the second coordinate position, the third coordinate position, and the fourth coordinate position.

2. The method for determining violations according to claim 1, characterized in that, The image data, comprising multiple frames, is processed to obtain alarm regions corresponding to the multiple frames, including: For each frame of the multi-frame image, the frame is divided based on the geometric center of each frame to obtain the divided multi-frame image, and each frame of the divided multi-frame image has multiple division regions. A calibration prompt message containing the divided multi-frame images is sent to a second object, and feedback information sent by the second object in response to the calibration prompt message is received. The feedback information includes the result of the second object calibrating the region type of multiple division regions of each frame of the divided multi-frame images. The region type includes: alarm region and non-alarm region. The region segmentation indicated in the feedback information as an alarm region is determined as the alarm region corresponding to multiple frames of images.

3. The method for determining violations according to claim 1, wherein the movement trajectory of the first object in the current frame image is determined based on the second coordinate position, the third coordinate position, and the fourth coordinate position, includes: Determine the first object feature when the fifth object is located at the fourth coordinate position, the second object feature corresponding to the first object being located at the second coordinate position, and the third object feature corresponding to the first object being located at the third coordinate position; If the feature overlap between the first object feature and the second object feature is greater than a preset value, and the feature overlap between the first object feature and the third object feature is greater than the preset value, then the fifth object is determined to be consistent with the first object, and the movement trajectory of the first object in the current frame image is determined according to the second coordinate position, the third coordinate position and the fourth coordinate position.

4. The method for determining violations according to claim 1, characterized in that, The positional difference between the second coordinate position and the third coordinate position is determined in the following manner: The second coordinate position corresponding to the first frame of the other frames and the second coordinate position corresponding to the last frame of the other frames are determined from the second coordinate position, and the position offset value between the second coordinate position corresponding to the first frame of the other frames and the second coordinate position corresponding to the last frame of the other frames is determined. The number of frames of the other frames in the multi-frame image is obtained, the ratio of the position offset to the number of frames is determined as the target second coordinate position, and the position difference between the target second coordinate position and the third coordinate position is determined as the position difference between the second coordinate position and the third coordinate position.

5. The method for determining violations according to claim 1, characterized in that, Determining the mapping relationship between the current frame image and the other frame images includes: If it is determined that a third object exists in both the current frame image and the other frame images, the first image feature of the third object in the current frame image and the second image feature of the third object in the other frame images are obtained; The transformation matrix between the first image feature and the second image feature is calculated according to a preset algorithm to obtain the mapping relationship between the first image feature and the second image feature.

6. The method for determining violations according to claim 1, characterized in that, Before processing the multiple frames of images included in the image data to obtain the movement trajectory of the first object within the target area, the method further includes: Determining the first object from the multiple frames of images, wherein determining the first object from the multiple frames of images includes: Obtain the different objects identified in the multi-frame images; Based on the object characteristics of the different objects, the identities of the different objects are identified to obtain the object types corresponding to the different objects; Obtain a fourth object of the same type from among the different objects. If the identity of the fourth object is the same as that of the first object, then identify the fourth object as the first object.

7. The method for determining violations according to claim 6, characterized in that, In the process of acquiring the different objects identified in the multiple frames of images, the method further includes: For each of the different objects, a first object area is determined for each object in advance, and a second object area of ​​each object in the multi-frame images is obtained; If the difference between the area of ​​the first object and the area of ​​the second object is less than or equal to a preset value, the object feature corresponding to the area of ​​the first object is determined as the object feature of each object.

8. The method for determining violations according to claim 1, characterized in that, Based on the relationship between the last trajectory position of the movement trajectory and the location distribution of the alarm area, determine whether the first object has engaged in any violations, including: From all the trajectory points included in the final trajectory position, obtain a preset number of continuous trajectory points, and obtain each trajectory ray of each of the continuous trajectory points at a preset angle; Given that each trajectory ray intersects with the boundary line of the alarm area, the number of intersection points between each trajectory ray and the boundary line of the alarm area is obtained. If the number of intersections between each trajectory ray and the boundary line of the alarm area meets a preset condition, and if the final trajectory position of the action trajectory is located within the alarm area, then it is determined that the first object has committed a violation.

9. The method for determining violations according to claim 1, characterized in that, Based on the relationship between the last trajectory position of the movement trajectory and the location distribution of the alarm area, determine whether the first object has engaged in any violations, including: From all the trajectory points included in the final trajectory position, obtain a preset number of continuous trajectory points, and obtain any trajectory ray of any trajectory point of the continuous trajectory points at a preset angle; If the number of intersections between any trajectory ray and the boundary line of the alarm area does not meet a preset condition, and the final trajectory position of the movement trajectory is not located within the alarm area, then it is determined that the first object has not committed any violation.

10. The method for determining violations according to claim 8, characterized in that, After determining that the first object has engaged in a violation, the method further includes: If the final trajectory position is located within the alarm area, an alarm message is sent to the first object.

11. A device for determining a violation, characterized in that, include: The acquisition module is used to acquire image data collected by the image acquisition device for a target area within a preset time period, wherein the image acquisition device is set at the bottom of the target device to acquire images of the area below the target device; The module is used to process the multiple frames of images included in the image data to obtain the alarm area corresponding to the multiple frames of images and the movement trajectory of the first object in the target area; The sending module is used to determine whether the first object has violated regulations based on the last trajectory position of the action trajectory and the location distribution relationship of the alarm area, wherein the violation refers to abnormal behavior associated with the target area; The sending module is further configured to: ... The aforementioned module is further configured to acquire the current frame image and other frame images preceding the current frame image from the multi-frame images, and determine the mapping relationship between the current frame image and the other frame images; acquire the first coordinate position of the first object in the other frame images, and determine the second coordinate position corresponding to the first coordinate position in the coordinate system of the current frame image according to the mapping relationship; acquire the third coordinate position of the first object in the coordinate system of the current frame image; determine the sum of the following parameters: the third coordinate position, the position difference between the second coordinate position and the third coordinate position; determine the fourth coordinate position in the coordinate system of the current frame image according to the sum; and determine the movement trajectory of the first object in the current frame image according to the second coordinate position, the third coordinate position, and the fourth coordinate position.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 10.

13. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 10 through the computer program.

Images