Image processing method and device based on edge access and storage medium

Abstract

The invention discloses an image processing method and device based on edge access and a storage medium. The method comprises the steps of obtaining video stream information of a target area of a production line; the video stream information comprises at least two frames of depth images of the target area; determining a fluctuation detection result used for representing the fluctuation condition of the depth data based on the depth data of the depth image and the depth data of the adjacent depth image; the adjacent depth image is a depth image adjacent to the depth image in image acquisition time; and determining whether the target area is abnormal or not based on the fluctuation detection result.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an edge access-based image processing method, device and storage medium. Background Art

[0002] With the development of society, security monitoring has gradually become a research hotspot, especially in manufacturing industries, such as production lines, where the demand for security monitoring has gradually become a top priority. Summary of the invention

[0003] In view of this, the embodiments of the present application at least provide an image processing method, device and storage medium based on edge access.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method based on edge access, the method comprising:

[0006] Acquire video stream information of a target area of ​​a production line; the video stream information includes at least two frames of depth images of the target area;

[0007] Determine a fluctuation detection result for characterizing fluctuation of the depth data based on the depth data of the depth image and the depth data of an adjacent depth image; the adjacent depth image is a depth image adjacent to the depth image in image acquisition time;

[0008] Based on the fluctuation detection results, determine whether there is an abnormality in the target area.

[0009] In a second aspect, an embodiment of the present application provides an edge access-based image processing device, the image processing device comprising:

[0010] An acquisition unit, used to acquire video stream information of a target area of ​​a production line; the video stream information includes at least two frames of depth images of the target area;

[0011] A first determining unit is used to determine a fluctuation detection result for characterizing a fluctuation condition of the depth data based on the depth data of the depth image and the depth data of an adjacent depth image; the adjacent depth image is a depth image that is adjacent to the depth image in image acquisition time;

[0012] The second determining unit is used to determine whether there is an abnormality in the target area based on the fluctuation detection result.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implement some or all of the steps in the above method when executed by a processor.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0018] Figure 1 Schematic diagram of an implementation process of an edge access-based image processing method provided in an embodiment of the present application Figure 1 ;

[0019] Figure 2 Schematic diagram of an implementation process of an edge access-based image processing method provided in an embodiment of the present application Figure 2 ;

[0020] Figure 3 Schematic diagram of an implementation process of an edge access-based image processing method provided in an embodiment of the present application Figure 3 ;

[0021] Figure 4 Schematic diagram of an implementation process of an edge access-based image processing method provided in an embodiment of the present application Figure 4 ;

[0022] Figure 5 Schematic diagram of an implementation process of an edge access-based image processing method provided in an embodiment of the present application Figure 5 ;

[0023] Figure 6 A schematic diagram of the structure of an edge access-based image processing device provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a hardware entity of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0026] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0029] With the development of society, security monitoring has gradually become a research hotspot, especially in the manufacturing industry, such as production lines, where the demand for security monitoring has gradually become a top priority. In related technologies, high-definition cameras are usually installed above the target area of ​​the production line to shoot videos in real time, and then the background monitoring center monitors or reviews the monitoring records afterwards. However, due to the long distance between the shooting location and the background monitoring center, wireless transmission is often performed through a cloud server, but the captured videos are often large files, which require a lot of network resources, and the transmission efficiency is very low and the transmission time is very long. At the same time, some related technologies use image model training methods to identify flames, which requires the collection of a large amount of data sets in the early stage, and there are many algorithm processes, which will occupy a lot of system computing power. At the same time, the monitoring results may still be inaccurate, and the image processing method cannot determine the size of the flame, and cannot determine the intensity of the flame. The intense burning of a small range of flames is still fatal to production lines or warehouses, because production lines or warehouses require more stringent security monitoring, such as the intensity of a small range of flames. The judgment of the degree may be caused by the deflagration of production equipment or flammable and explosive materials. The relevant technology only monitors the fire area, which is obviously unable to monitor the intensity of the flame when the fire area is certain. Moreover, the production line is long and a large number of monitoring points need to be deployed. The hardware and system research and development costs of safety monitoring are high, and the enterprise benefits are low. Therefore, how to use the edge computing gateway based on edge access to selectively process and generate images that need to assist in confirming whether a fire has occurred, thereby improving the transmission efficiency of monitoring resources, saving transmission time and video confirmation fire alarm interaction time, saving system computing power, and eliminating the data collection steps for model training, thereby improving the efficiency of safety monitoring of the production line, is a problem that needs to be urgently solved in related technologies.

[0030] In order to solve the above technical problems, the embodiment of the present application provides an image processing method based on edge access, which can be applied to an edge computing gateway. Figure 1 As shown, the above processing method can be implemented through steps S101 to S103:

[0031] Step S101, obtaining video stream information of a target area of ​​a production line; the video stream information includes at least two frames of depth images of the target area.

[0032] Here, the target area can be any area of ​​the production line. In some embodiments, the target area can also be an area in the production line where an abnormality is prone to occur. Exemplarily, the abnormality can be a fire.

[0033] In an embodiment of the present application, a depth camera is provided at the target area of ​​the production line, through which at least two frames of depth images of the target area can be collected, thereby forming video stream information.

[0034] In some embodiments, the depth camera may select a preset number of depth images according to the image acquisition order to form a video stream and send it to the edge computing gateway. Exemplarily, the preset number may be 60.

[0035] Step S102, determining a fluctuation detection result for characterizing the fluctuation of the depth data based on the depth data of the depth image and the depth data of an adjacent depth image; the adjacent depth image is a depth image adjacent to the depth image in image acquisition time.

[0036] Here, the fluctuation of the depth data refers to the fluctuation of the depth data between pixels at the same position of two adjacent frames of depth images at the time of image acquisition.

[0037] It is understandable that, because at least two depth images are acquired by the same depth camera, the size of each depth image is consistent, and therefore each depth image includes the same number of pixels. Therefore, it is possible to compare the fluctuation of depth data between pixels at the same position of two adjacent depth images at the image acquisition time.

[0038] In some embodiments, for two depth images that are adjacent in image acquisition time, such as a first depth image and a second depth image, the fluctuation of depth data between all pixels in the first depth image and all pixels in the second depth image can be compared simultaneously. In other embodiments, the comparison can also be performed in sequence according to the two-dimensional coordinate information of the pixels.

[0039] In some embodiments, in order to reduce the amount of computation of the edge computing gateway, each depth image can be segmented to obtain depth sub-images, and the depth data of each adjacent depth sub-image can be compared.

[0040] Step S103: Determine whether there is any abnormality in the target area based on the fluctuation detection result.

[0041] In the embodiment of the present application, when the fluctuation detection result indicates that the depth data fluctuates more violently, it can be determined that there is an abnormality in the target area; when the fluctuation detection result indicates that the depth data fluctuates less violently, it can be determined that there is no abnormality in the target area. Among them, whether the fluctuation of the depth data is violent can be determined based on the fluctuation change amount of the depth data. When the fluctuation change amount of the depth data is greater than the preset change amount, it is determined that the fluctuation of the depth data is violent. In some embodiments, it is also possible to determine whether the depth data fluctuates violently based on the number of pixels in which the depth data has changed. When the number of pixels in which the depth data has changed is greater than the preset number, it is determined whether the depth data has fluctuated violently.

[0042] It is understandable that whether the target area has an abnormality is related to whether the depth data of the pixels of the depth image fluctuates. For example, when the abnormality of the target area is a fire, the particularity of the flame will affect the fluctuation of the depth data. Therefore, it can be determined whether the target area has an abnormality through the fluctuation detection result used to characterize the fluctuation of the depth data.

[0043] In an embodiment of the present application, the fluctuation detection result used to characterize the fluctuation of the depth data is determined by the depth data of the depth image and the depth data of the adjacent depth image; then, based on the fluctuation detection result, it is determined whether there is an abnormality in the target area. In this way, compared with the related art of identifying whether an abnormality occurs in the production line by means of an image model, the embodiment of the present application can accurately determine whether an abnormality occurs in the target area and the degree of the abnormality through the fluctuation of the depth data, thereby improving the accuracy of identifying whether the production line is abnormal. In addition, the embodiment of the present application performs abnormality identification through an edge computing gateway, and because the edge computing gateway is closer to the production line, the efficiency of video stream information transmission can be improved.

[0044] In some embodiments, Figure 2 As shown, the above step S102 can be implemented through step S201 and step S202; the above step S103 can be implemented through step S203:

[0045] Step S201, segmenting each depth image to obtain at least one depth sub-image of each depth image; different depth sub-images in each depth image have different image coding information, and in each depth image, the image coding information of depth sub-images at the same position of the depth image is the same.

[0046] In an embodiment of the present application, each depth image may be segmented according to a preset number of rows and a preset number of columns, so that each depth image includes at least one depth sub-image having a preset number of rows and a preset number of columns. Exemplarily, the preset number of rows and the preset number of columns may both be 20, that is, the number of pixel rows and the number of pixel columns of each depth sub-image of each depth image are both 20.

[0047] In an embodiment of the present application, after the depth image is segmented, each depth sub-image of each depth image can be numbered so that different depth sub-images in each depth image have different image coding information, and in each depth image, the image coding information of the depth sub-images at the same position of the depth image is the same.

[0048] Exemplarily, the video stream information includes 3 depth images, namely depth image 1, depth image 2 and depth image 3, each depth image includes 2 depth sub-images, the image encoding information of the first depth sub-image may be (1, 1), and the image encoding information of the second depth sub-image may be (1, 2). That is, depth image 1, depth image 2 and depth image 3 all include a depth sub-image of (1, 1) and a depth sub-image of (1, 2).

[0049] Step S202, for each image coding information, based on the depth data of the depth image and the depth data of the adjacent depth images, determine whether there is at least one target depth sub-image corresponding to the image coding information in at least two depth sub-images; the target depth sub-image is an image whose depth data fluctuation degree is greater than a preset value.

[0050] Here, the target depth sub-image may be a depth sub-image having drastic depth data fluctuations among at least one depth sub-image in a depth image.

[0051] In the embodiment of the present application, the target depth sub-image corresponding to each image coding information may be determined in sequence according to the order of the image coding information.

[0052] Exemplarily, the video stream information includes 3 frames of depth images, namely depth image 1, depth image 2 and depth image 3; depth image 1, depth image 2 and depth image 3 all include a depth sub-image of (1, 1) and a depth sub-image of (1, 2). The target depth sub-image corresponding to (1, 1) can be determined based on the depth data of the depth sub-image of (1, 1) of depth image 1, the depth data of the depth sub-image of (1, 1) of depth image 2, the depth data of the depth sub-image of (1, 1) of depth image 2, and the depth data of the depth sub-image of (1, 1) of depth image 3; and then the target depth sub-image corresponding to (1, 2) can be determined according to the above method.

[0053] In some embodiments, the target depth sub-image corresponding to each image encoding information may be determined simultaneously.

[0054] Step S203 : determining whether there is an abnormality in the target area based on whether there is at least one target depth sub-image corresponding to the image coding information in the at least two depth sub-images.

[0055] In an embodiment of the present application, when there is at least one target depth sub-image corresponding to the image coding information in at least two depth sub-images, it can be determined that there is an abnormality in the target area; when there is no at least one target depth sub-image corresponding to the image coding information in at least two depth sub-images, it can be determined that there is no abnormality in the target area.

[0056] In the embodiment of the present application, as long as the target depth sub-image exists in the depth sub-image corresponding to any image coding information, it is determined that an abnormality exists in the target area.

[0057] In some embodiments, Figure 3 As shown, the above step S202 of "determining whether there is at least one target depth sub-image corresponding to the image coding information in at least two depth sub-images based on the depth data of the depth image and the depth data of the adjacent depth image" can be implemented by steps S301 to S303:

[0058] Step S301, determining at least one depth sub-image pair in at least one depth sub-image corresponding to at least two frames of depth images; the image coding information of different depth sub-images of each depth sub-image pair is the same, and the depth images corresponding to different depth sub-images are adjacent in image acquisition time.

[0059] Here, the image coding information of different depth sub-images of the depth sub-image pair is the same, and the depth images corresponding to the different depth sub-images are adjacent in image acquisition time.

[0060] Exemplarily, the video stream information includes three frames of depth images, namely depth image 1, depth image 2 and depth image 3; depth image 1, depth image 2 and depth image 3 all include depth sub-image (1, 1) and depth sub-image (1, 2). Because depth image 1 is adjacent to depth image 2, and depth image 2 is adjacent to depth image 2; therefore, the depth sub-image pair corresponding to the image coding information (1, 1) includes two, namely: the depth sub-image (1, 1) of depth image 1 and the depth sub-image (1, 1) of depth image 2, and the depth sub-image (1, 1) of depth image 2 and the depth sub-image (1, 1) of depth image 3; the depth sub-image pair corresponding to the image coding information (1, 2) also includes two, namely: the depth sub-image (1, 2) of depth image 1 and the depth sub-image (1, 2) of depth image 2, and the depth sub-image (1, 2) of depth image 2 and the depth sub-image (1, 2) of depth image 3.

[0061] Step S302: Determine a change in depth data of at least one depth sub-image pair.

[0062] In the embodiment of the present application, for each depth sub-image pair, it can be determined whether the depth data of the pixels at the corresponding positions of each depth sub-image pair has changed.

[0063] Exemplarily, for a depth sub-image pair including a depth sub-image at (1, 1) of depth image 1 and a depth sub-image at (1, 1) of depth image 2, both depth sub-images include 3 pixels, determine whether the depth data of pixel 1 in the depth sub-image at (1, 1) of depth image 1 and the depth data of pixel 1 in the depth sub-image at (1, 1) of depth image 2 have changed.

[0064] Step S303 : determining whether there is at least one target depth sub-image corresponding to the image coding information based on the change conditions corresponding to the at least one depth sub-image pair.

[0065] In the embodiment of the present application, it can be determined whether there is at least one target depth sub-image corresponding to the image encoding information according to the number of pixel values ​​whose depth data has changed.

[0066] Exemplarily, for a depth sub-image pair including a depth sub-image at (1, 1) of depth image 1 and a depth sub-image at (1, 1) of depth image 2, both depth sub-images include 3 pixels, and the depth data of the 3 pixels have changed. It can be determined that there is a target depth sub-image in the depth sub-image at (1, 1) of depth image 1 and the depth sub-image at (1, 1) of depth image 2.

[0067] In some embodiments, the above step S302 may be implemented by steps S3021 and S3022:

[0068] Step S3021 : for each depth sub-image pair, compare depth data between two pixels of a pixel pair of the depth sub-image pair; the pixels in the pixel pair are at the same pixel position in the two depth sub-images of the depth sub-image pair.

[0069] Here, the two pixels in the pixel pair belong to different depth sub-images, the different depth sub-images are a depth sub-image pair, and the two pixels are at the same pixel position in the depth sub-images, where the pixel position refers to the position of the pixel in the image coordinate system of the depth sub-image.

[0070] Exemplarily, for a depth sub-image pair including a depth sub-image at (1, 1) of depth image 1 and a depth sub-image at (1, 1) of depth image 2, the depth sub-image at (1, 1) of depth image 1 includes pixel 1 with pixel coordinates (1, 1), and the depth sub-image at (1, 1) of depth image 2 includes pixel 2 with pixel coordinates (1, 1), then pixel 1 and pixel 2 can be regarded as a pixel pair.

[0071] In the embodiment of the present application, the depth data between two pixels of the pixel pair of the depth sub-image pair can be compared to see whether they have changed. Here, the depth data changes, which can be negative changes or positive changes, that is, the depth data increases or decreases over time.

[0072] Step S3022 : for each depth sub-image pair, determine the number of pixel pairs whose depth data of the depth sub-image pair has changed as the change of the depth data of the depth sub-image pair.

[0073] In an embodiment of the present application, it is possible to determine whether the depth data between two pixels in a pixel pair in each depth sub-image pair has changed. If a change has occurred, the pixel pair is determined as a pixel pair whose depth data has changed. Then, the number of all pixel pairs whose depth data has changed in each depth sub-image pair is calculated, and the change in the depth data of the depth sub-image pair is determined based on the number.

[0074] Exemplarily, for a depth sub-image pair including a depth sub-image at (1, 1) of depth image 1 and a depth sub-image at (1, 1) of depth image 2, both depth sub-images include 3 pixels, that is, the depth sub-image pair includes three pixel pairs, and then the number of pixel pairs whose depth data changes among the three pixel pairs is determined.

[0075] In some embodiments, the total number of pixel pairs whose depth data of the depth sub-image pair has changed may be determined as the change in the depth data of the depth sub-image pair. In other embodiments, when the number of pixel pairs whose depth data has changed includes a first number of pixel pairs whose depth data has changed negatively and a second number of pixel pairs whose depth data has changed positively, the first number and the second number may be determined as the change in the depth data of the depth sub-image pair.

[0076] Exemplarily, for a depth sub-image pair including a depth sub-image of depth image 1 (1, 1) and a depth sub-image of depth image 2 (1, 1), both depth sub-images include 3 pixels, that is, the depth sub-image pair includes three pixel pairs. If the number of pixel pairs with negative changes in depth data among the three pixel pairs is 1, and the number of pixel pairs with positive changes in depth data is 2; when the change in depth data of the depth sub-image pair is the total number of pixel pairs with changed depth data of the depth sub-image pair, 3 is used as the change in depth data of the depth sub-image pair; when the change in depth data of the depth sub-image pair is the first number and the second number, 1 and 2 are used as the change in depth data of the depth sub-image pair.

[0077] In some embodiments, the number of pixel pairs whose depth data has changed includes a first number of pixel pairs whose depth data has changed negatively, and a second number of pixel pairs whose depth data has changed positively; the above step S303 can be implemented by step S3031:

[0078] Step S3031 , when the first number and the second number are both greater than a first preset threshold, determining a depth sub-image with a later image acquisition time in the depth sub-image pair as a target depth sub-image.

[0079] Exemplarily, the first preset threshold may be 10.

[0080] It can be understood that when the number of pixel pairs whose depth data has changed, including the first number of pixel pairs whose depth data has changed negatively, and the second number of pixel pairs whose depth data has changed positively, are both greater than the first preset threshold, it indicates that the abnormal object in the depth sub-image is undergoing a drastic change. For example, the abnormal object is a flame, because during the combustion process, the shape, size and intensity of the flame may change with changes in conditions such as fuel supply, oxygen concentration and wind speed, which is reflected in the image as the depth data of the flame changing drastically over time. Therefore, when both the first number and the second number are greater than the first preset threshold, it is determined that the target depth sub-image exists in the depth sub-image pair.

[0081] In the embodiment of the present application, the depth sub-image with a later image acquisition time in the depth sub-image pair can be determined as the target depth sub-image. It can be understood that, because it has been determined that there is a target depth sub-image in the depth sub-image pair, in order to obtain the latest state of the abnormal object in the depth sub-image, the depth sub-image with a later image acquisition time in the depth sub-image pair can be determined as the target depth sub-image.

[0082] In some embodiments, Figure 4 As shown, the above step S103 can be implemented through step S401; when the abnormality in the target area is a fire, the above image recognition method can also be implemented through steps S402 to S404:

[0083] Step S401 : when a target depth sub-image exists in the depth sub-images of at least two frames of depth images, it is determined that an abnormality exists in the target area.

[0084] It is understandable that when there is a target depth sub-image in the depth sub-image, it means that the depth data of the abnormal object in the depth sub-image fluctuates dramatically over time, and it can be determined that there is an abnormality in the target area. For example, the abnormality in the target area may be that a fire is occurring in the target area.

[0085] Step S402: determining the location information of the fire and the flame area information in the target area based on the target depth sub-image.

[0086] In the embodiment of the present application, the edge information of the flame in the target area can be determined by an image edge recognition algorithm, and then the area information of the flame in the target depth sub-image (ie, the flame area information) can be determined based on the edge information of the flame.

[0087] In some embodiments, the homography matrix between the image coordinate system and the real geographic coordinate system can be determined through the attribute information of the depth camera. Then, the position information of the flame in the target area of ​​the production line in the target depth sub-image is determined through the homography matrix. Exemplarily, the attribute information of the depth camera may include: camera intrinsic parameters, camera extrinsic parameters, pan / tilt angle, camera installation height, camera installation longitude, and camera installation latitude of the depth camera.

[0088] Step S403: determining the flame severity level of the fire based on the first quantity and the second quantity.

[0089] In the embodiment of the present application, the first number and the second number are proportional to the severity of the fire flame. That is, the larger the first number and the second number are, the greater the severity of the fire flame, that is, the more severe the fire is.

[0090] Step S404: output the target depth sub-image, the location information of the fire in the target area, the flame area information, and the flame intensity level of the fire.

[0091] In an embodiment of the present application, the edge computing gateway can output the target depth sub-image, the location information of the fire in the target area, the flame area information, and the flame intensity level of the fire to a monitoring background that is communicated with the edge computing gateway. The monitoring background can determine the corresponding execution strategy based on the target depth sub-image, the location information of the fire in the target area, the flame area information, and the flame intensity level of the fire.

[0092] In some embodiments, Figure 5 As shown, the above image recognition method can also be implemented through steps S501 to S505:

[0093] Step S501: Acquire at least one target depth sub-image corresponding to each image coding information.

[0094] Step S502: based on the image acquisition time, merge the target depth sub-images corresponding to the adjacent image encoding information to obtain a merged image.

[0095] In an embodiment of the present application, for each image acquisition time of at least two frames of depth images, a target depth sub-image corresponding to the image acquisition time can be obtained from at least one target depth sub-image corresponding to each image coding information, and then the target depth sub-images corresponding to adjacent image coding information can be merged together.

[0096] Exemplarily, the target depth sub-image for image coding information (1, 1) includes target depth sub-image 1 of depth image 1, and target depth sub-image 2 of depth image 2; the target depth sub-image for image coding information (1, 2) includes target depth sub-image 3 of depth image 1, and target depth sub-image 4 of depth image 2; for the image acquisition time of depth image 1, the corresponding target depth sub-images include target depth sub-image 1 and target depth sub-image 3, the image coding information of target depth sub-image 1 is (1, 1), and the image coding information of target depth sub-image 3 is (1, 2). Since (1, 1) and (1, 2) are adjacent, target depth sub-image 1 and target depth sub-image 3 can be merged together to obtain a merged image.

[0097] In some embodiments, to determine whether two pieces of image coding information are adjacent, the Euclidean distance between the two pieces of image coding information may be calculated. If the Euclidean distance between the two pieces of image coding information is less than a preset distance, it is determined that the two pieces of image coding information are adjacent.

[0098] Step S503: determining the actual coordinate information of each vertex based on the image coordinate information of each vertex of the merged image.

[0099] In the embodiment of the present application, the actual coordinate information of each vertex can be determined based on the above homography matrix and the image coordinate information of each vertex.

[0100] Step S504: based on the actual coordinate information of each vertex, determine the actual area information corresponding to the depth data with a fluctuation degree greater than a preset value in the merged image.

[0101] In an embodiment of the present application, the polygon enclosed by each vertex can be determined based on the actual coordinate information of each vertex, and then the area of ​​the polygon can be determined as the actual area information corresponding to the depth data in the merged image whose fluctuation degree is greater than a preset value.

[0102] Step S505, outputting the actual coordinate information, actual area information and merged image of each vertex.

[0103] In an embodiment of the present application, the edge computing gateway can output the actual coordinate information, actual area information and merged image of each vertex to a monitoring background that is communicated with the edge computing gateway. The monitoring background can determine the corresponding execution strategy based on the actual coordinate information, actual area information and merged image of each vertex.

[0104] In some embodiments, the above image recognition method can also be implemented through steps S601 to S602:

[0105] Step S601 : performing difference processing on depth data of every two depth images that are adjacent in image acquisition time in at least two frames of depth images.

[0106] Step S602 , removing depth images whose difference processing results indicate that the depth data has not changed from at least two depth image frames, to obtain a processed depth image; the processed depth image is used to determine the fluctuation detection result.

[0107] In an embodiment of the present application, in order to reduce the amount of calculation required by the edge computing gateway to determine whether an abnormality occurs in the target area, the depth data of every two depth images that are adjacent in image acquisition time can be differentially processed first, and the depth images whose depth data have not changed can be eliminated.

[0108] In the embodiment of the present application, after obtaining the depth image after elimination, a fluctuation detection result for characterizing the fluctuation of the depth data can be determined through the depth data of the processed depth image and the depth data of the processed adjacent depth images.

[0109] With the development of society, security monitoring has gradually become a research hotspot, especially in manufacturing industries, such as production lines, where the demand for security monitoring has gradually become a top priority.

[0110] In the prior art, a high-definition camera is usually installed above the target area of ​​the production line to shoot video in real time, and then the back-end monitoring center monitors or reviews the monitoring records afterwards. Due to the long distance between the shooting location and the back-end monitoring center, wireless transmission is often performed through a cloud server, but the video files are often large, requiring a large amount of network resources, and the transmission efficiency is very low and the transmission time is very long.

[0111] At the same time, on the one hand, the existing technology of identifying flames through image model training requires the collection of a large amount of data sets in the early stage, and there are many algorithm processes, which will take up a lot of system computing power. On the other hand, the monitoring results may still be inaccurate, and the image processing method cannot determine the size of the flame or the intensity of the flame. The intense burning of a small-scale flame is still fatal to the production line or warehouse. This is because the production line or warehouse requires stricter safety monitoring. For example, the judgment of the intensity of a small-scale flame may be caused by the deflagration of production equipment or flammable and explosive materials.

[0112] However, the existing technology only monitors the fire area, which obviously cannot monitor the intensity of the flame when the fire area is certain. Moreover, the production line is long and a large number of monitoring points need to be deployed. The hardware and system development costs of safety monitoring are high, and the enterprise benefits are low. Therefore, how to selectively process and generate images that need to assist in confirming whether a fire has occurred based on edge access and using edge computing gateways, thereby improving the transmission efficiency of monitoring resources, saving transmission time and video confirmation of fire alarm interaction time, saving system computing power, and eliminating the data collection steps for model training, thereby improving the efficiency of production line safety monitoring work, is a problem that the existing technology urgently needs to solve. Therefore, the existing technology urgently needs a production line safety monitoring solution based on edge access.

[0113] Based on this, the present application provides a production line safety monitoring method based on edge access, the method comprising:

[0114] S100 , using a depth camera to capture depth images of a target area of ​​a production line at preset time intervals to form a video stream (the video stream information in the above embodiment).

[0115] It can be understood that before step S100, it includes: pre-setting a preset time interval, a preset number, a first preset threshold, a second preset threshold, a third preset threshold, a fourth preset threshold, a preset number of rows, a preset number of columns, camera intrinsic parameters of the depth camera, camera extrinsic parameters, gimbal angle, camera installation height, camera installation longitude, and camera installation latitude in the control module.

[0116] It can be understood that the camera intrinsic parameters, camera extrinsic parameters, gimbal angle, camera installation height, camera installation longitude, and camera installation latitude of the depth camera can be specifically set according to actual needs, the height of buildings around the target area, the camera model, and the relative position of the on-site depth camera installation position and the target area. This application does not specifically set the camera intrinsic parameters, camera extrinsic parameters, gimbal angle, camera installation height, camera installation longitude, and camera installation latitude of the depth camera. There is no regularity in setting the specific values, as long as it is applicable to the edge access-based production line safety monitoring method proposed in this application.

[0117] It can be understood that the preset number of rows, preset number of columns, preset quantity, preset time interval, first preset threshold, second preset threshold, third preset threshold, and fourth preset threshold can be set according to the needs of the users of this application, and there is no regularity in setting the specific numerical values, as long as it is applicable to the edge access-based production line safety monitoring method proposed in this application.

[0118] In some embodiments, the present application preferably sets the preset time interval to 1 second, the present application preferably sets the first preset threshold to 10, the present application preferably sets the second preset threshold to 1, the present application sets the third preset threshold to 30, and the present application sets the fourth preset threshold to 20. The above settings are obtained by the technical personnel of the present application through a large number of tests, and can well implement the edge access-based production line safety monitoring method described in the present application.

[0119] In some embodiments, the present application preferably sets the preset number of rows to 20 rows, and the present application preferably sets the preset number of columns to 20 columns. The above settings are obtained by the technical personnel of the present application through a large number of tests, and can well implement the edge access-based production line safety monitoring method described in the present application.

[0120] In some embodiments, the present application sets the preset number to 60.

[0121] S200. Starting from the current frame, a preset number of depth images are sequentially selected from the depth images in the video stream, and the depth images are sent to the edge computing gateway. The selected depth images are differentially processed to eliminate the image parts whose corresponding depth data have not changed in the selected depth images to obtain a preset number of target depth images (the processed depth images in the above embodiment). The preset number of target depth images are divided into sub-images with a preset number of rows and a preset number of columns (depth sub-images in the above embodiment), and the sub-images of each target depth image are numbered with two-dimensional coordinates according to the number of rows and columns (image encoding information in the above embodiment). The depth data of all pixel coordinates of the sub-images corresponding to the same two-dimensional coordinate encoding in two adjacent target depth images are traversed in sequence. If the depth data corresponding to the same pixel coordinate increases, the first counter is controlled to increase by one (the second number in the above embodiment); if the depth data corresponding to the same pixel coordinate decreases, the second counter is controlled to increase by one (the first number in the above embodiment).

[0122] It should be noted here that the control module includes an MCU control chip, and the MCU control chip has a first counter and a second counter built therein.

[0123] S300. After traversing all sub-images corresponding to the same two-dimensional coordinate code, determine whether the values ​​of the first counter and the second counter are both greater than the first preset threshold value, determine whether the depth data of the sub-image fluctuates violently based on the judgment result, determine whether a fire has occurred based on the judgment result, calculate the location of the fire, and determine the severity of the flame based on the values ​​of the first counter and the second counter, and send the sub-image, the location of the fire and the severity of the flame to the staff of the monitoring background through the cloud server.

[0124] Specifically, the determining whether the value of the first counter and the value of the second counter are both greater than a first preset threshold, and determining whether the depth data of the sub-image fluctuates violently according to the determination result, includes:

[0125] If the value of the first counter and the value of the second counter are both greater than the first preset threshold, it is determined that the depth data of the sub-image fluctuates violently (ie, the target depth sub-image in the above embodiment).

[0126] Specifically, the determining whether the value of the first counter and the value of the second counter are both greater than a first preset threshold, and determining whether the depth data of the sub-image fluctuates violently according to the determination result, further includes:

[0127] If the value of the first counter or the value of the second counter is less than or equal to the first preset threshold, it is determined that the depth data of the sub-image does not fluctuate violently.

[0128] It can be understood that if the value of the first counter and the value of the second counter are both greater than the first preset threshold, in the present application, that is, the value of the first counter and the value of the second counter are both greater than 10, it proves that the depth data of the sub-image fluctuates frequently within 60 seconds, which is similar to the characteristics of the flame, then it is determined that the depth data of the sub-image fluctuates violently, otherwise, it is determined that the depth data of the sub-image does not fluctuate violently.

[0129] In some embodiments, the determining, based on the determination result, whether to send the sub-image to a monitoring backend staff member through a cloud server to confirm whether a fire has occurred includes:

[0130] If it is determined that the depth data of the sub-image fluctuates violently, the sub-image is sent to the monitoring backend staff through the cloud server to confirm whether a fire has occurred.

[0131] In some embodiments, the determining, based on the determination result, whether to send the sub-image to a staff member of a monitoring backend through a cloud server to confirm whether a fire has occurred further includes:

[0132] If it is determined that the depth data of the sub-image does not fluctuate violently, the sub-image will not be sent to the monitoring backend staff through the cloud server to confirm whether a fire has occurred.

[0133] In some embodiments, the method further comprises:

[0134] After determining whether the depth data of the sub-image fluctuates violently according to the judgment result, the first counter and the second counter are cleared, and the depth data of the sub-image corresponding to the next two-dimensional coordinate encoding is continuously traversed to determine whether it fluctuates violently, and then all sub-images with violent depth data fluctuations are selected, and two sub-images are randomly selected therefrom, and it is determined whether the Euclidean distance of the two-dimensional coordinate encodings of the two sub-images is less than or equal to the second preset threshold value, and it is determined whether to merge the two sub-images (the merged image in the above embodiment) according to the judgment result, and the remaining sub-images are traversed in sequence according to the above method, and the sub-images whose Euclidean distance of the two-dimensional coordinate encodings is less than or equal to the second preset threshold value are merged, and the merged sub-images are recorded as merged images with violent depth data fluctuations, and then at least one merged image with violent depth data fluctuations is obtained. Select all pixel points of the four vertices on the boundary of the merged image where the depth data fluctuates violently, calculate the actual geographic coordinates corresponding to the four selected vertices based on the pixel point coordinates of the selected four vertices (the actual coordinate information in the above embodiment), calculate the actual geographic area where the depth data fluctuates violently based on the actual geographic coordinates corresponding to the four vertices, and send the merged images where the depth data fluctuates violently and the actual geographic coordinates corresponding to the pixel points on their selected boundaries (the actual coordinate information of each vertex of the merged image in the above embodiment) and the actual geographic area where the depth data fluctuates violently (the actual area information corresponding to the depth data in the merged image in the above embodiment whose fluctuation degree is greater than a preset value) to the staff of the monitoring background through the cloud server to confirm whether a fire has occurred.

[0135] It can be understood that the present application further saves the processing time of the staff for self-query and calculation of related data by sending the actual geographic coordinates corresponding to the pixel points on the selected boundary and the actual geographic area with severe depth data fluctuations through the cloud server to the staff of the monitoring background, further improves the transmission efficiency of monitoring resources, further saves transmission time and time for video confirmation of fire alarm interaction, further improves the efficiency of fire monitoring work, and greatly improves the intelligence and usability of the present application.

[0136] In some embodiments, determining whether the Euclidean distance between the two-dimensional coordinate codes of the two sub-images is less than or equal to a second preset threshold, and determining whether to merge the two sub-images according to the determination result, includes:

[0137] If the Euclidean distance between the two-dimensional coordinate codes of the two sub-images is less than or equal to a second preset threshold, it is determined whether to merge the two sub-images.

[0138] Specifically, the determining whether the Euclidean distance of the two-dimensional coordinate codes of the two sub-images is less than or equal to a second preset threshold, and determining whether to merge the two sub-images according to the determination result, further includes:

[0139] If the Euclidean distance between the two-dimensional coordinate codes of the two sub-images is greater than a second preset threshold, it is determined whether to merge the two sub-images.

[0140] It can be understood that if the Euclidean distance of the two-dimensional coordinate encodings of the two sub-images is less than or equal to the second preset threshold (the preset distance in the above embodiment), it proves that the two sub-images belong to the same connected domain in the target depth image. Therefore, the two sub-images can be merged, thereby realizing the intelligent merging of images that need auxiliary confirmation of whether a fire has occurred, further improving the transmission efficiency of monitoring resources, further saving transmission time and video confirmation fire alarm interaction time, further improving the efficiency of fire monitoring work, and further improving the intelligence level of this application.

[0141] In some embodiments, determining the severity level of the flame according to the values ​​of the first counter and the second counter includes:

[0142] Determine whether any one of the values ​​of the first counter and the second counter is greater than or equal to the third preset threshold value. If so, determine that the intensity level of the flame is level three; if not, determine whether any one of the values ​​of the first counter and the second counter is greater than or equal to the fourth preset threshold value. If so, determine that the intensity level of the flame is level two; if not, determine that the intensity level of the flame is level one.

[0143] It should be noted here that the intensity of the flame is divided into three levels: three levels, two levels and one level from large to small.

[0144] In some embodiments, calculating the actual geographic coordinates corresponding to the four selected vertices according to the pixel coordinates of the four selected vertices includes:

[0145] Get the camera intrinsic parameters, camera extrinsic parameters, gimbal angle, camera installation height, camera installation longitude, and camera installation latitude of the depth camera, calculate the homography matrix for pixel coordinate and geographic coordinate conversion based on the camera intrinsic parameters, camera extrinsic parameters, gimbal angle, camera installation height, camera installation longitude, and camera installation latitude of the depth camera, and calculate the actual geographic coordinates corresponding to the selected four vertices based on the homography matrix.

[0146] Figure 6 A schematic diagram of the structure of an edge access-based image processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the edge access-based image processing device 600 includes: an acquisition unit 601, a first determination unit 602, and a second determination unit 603, wherein:

[0147] An acquisition unit 601 is used to acquire video stream information of a target area of ​​a production line; the video stream information includes at least two frames of depth images of the target area;

[0148] A first determining unit 602 is used to determine a fluctuation detection result for characterizing a fluctuation of the depth data based on the depth data of the depth image and the depth data of an adjacent depth image; the adjacent depth image is a depth image that is adjacent to the depth image in image acquisition time;

[0149] The second determination unit 603 is used to determine whether there is an abnormality in the target area based on the fluctuation detection result.

[0150] In some embodiments, the first determination unit 602 is also used to perform segmentation processing on each depth image to obtain at least one depth sub-image of each depth image; different depth sub-images in each depth image have different image coding information, and in each depth image, the image coding information of the depth sub-images at the same position of the depth image is the same; for each image coding information, based on the depth data of the depth image and the depth data of the adjacent depth images, determine whether there is at least one target depth sub-image corresponding to the image coding information in at least two depth sub-images; the target depth sub-image is an image whose depth data fluctuation degree is greater than a preset value.

[0151] In some embodiments, the first determination unit 602 is further used to determine at least one depth sub-image pair in at least one depth sub-image corresponding to at least two frames of depth images; the image coding information of different depth sub-images of each depth sub-image pair is the same, and the depth images corresponding to different depth sub-images are adjacent in image acquisition time; determine the change of depth data of at least one depth sub-image pair; and determine whether there is at least one target depth sub-image corresponding to the image coding information based on the change corresponding to at least one depth sub-image pair.

[0152] In some embodiments, for each depth sub-image pair, the first determination unit 602 is further used to compare the depth data between two pixels of a pixel pair of the depth sub-image pair; the pixels in the pixel pair are at the same pixel position in the two depth sub-images of the depth sub-image pair; and the number of pixel pairs whose depth data of the depth sub-image pair has changed is determined as the change in the depth data of the depth sub-image pair.

[0153] In some embodiments, the number of pixel pairs whose depth data has changed includes a first number of pixel pairs whose depth data has undergone negative changes, and a second number of pixel pairs whose depth data has undergone positive changes; the first determination unit 602 is also used to determine, for each depth sub-image pair, based on the changes corresponding to at least one depth sub-image pair, whether there is at least one target depth sub-image corresponding to the image encoding information, including: when the first number and the second number are both greater than a first preset threshold, determining the depth sub-image in the depth sub-image pair with a later image acquisition time as the target depth sub-image.

[0154] In some embodiments, the second determination unit 603 is also used to determine that there is an abnormality in the target area when there is a target depth sub-image in the depth sub-images of at least two frames of depth images; the above-mentioned edge access-based image processing device 600 also includes a fire determination unit: the fire determination unit is used to determine the location information and flame area information of the fire in the target area based on the target depth sub-image; determine the flame intensity level of the fire based on the first quantity and the second quantity; output the target depth sub-image, the location information of the fire in the target area, the flame area information and the flame intensity level of the fire.

[0155] In some embodiments, the above-mentioned edge access-based image processing device 600 also includes a merging unit; the merging unit is used to obtain at least one target depth sub-image corresponding to each image coding information; based on the image acquisition time, the target depth sub-images corresponding to adjacent image coding information are merged to obtain a merged image; based on the image coordinate information of each vertex of the merged image, the actual coordinate information of each vertex is determined; based on the actual coordinate information of each vertex, the actual area information corresponding to the depth data in the merged image whose fluctuation degree is greater than a preset value is determined; and the actual coordinate information, actual area information and merged image of each vertex are output.

[0156] In some embodiments, the above-mentioned edge access-based image processing device 600 also includes a preprocessing unit; the preprocessing unit is used to perform differential processing on the depth data of every two depth images that are adjacent in image acquisition time in at least two frames of depth images; eliminate the depth images whose differential processing results represent that the depth data has not changed in at least two frames of depth images, to obtain a processed depth image; the processed depth image is used to determine the fluctuation detection result.

[0157] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0158] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0159] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0160] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0161] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps in the above method.

[0162] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0163] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0164] Figure 7 is a hardware entity schematic diagram of an electronic device in an embodiment of the present application, such as Figure 7 As shown, the hardware entity of the electronic device 700 includes: a processor 701, a communication interface 702 and a memory 703, wherein:

[0165] The processor 701 generally controls the overall operation of the electronic device 700 , and the overall operation may be to implement the processing method provided in the embodiment of the present application.

[0166] The communication interface 702 enables the computer device to communicate with other terminals or servers through a network.

[0167] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or processed by the processor 701 and each module in the electronic device 700 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 701, the communication interface 702, and the memory 703 through the bus 704.

[0168] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the processing method of any of the above embodiments.

[0169] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0170] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0171] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0172] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0173] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0174] The above are only implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. An image processing method based on edge access, the method being applied to an edge computing gateway; the method comprising: Obtain video stream information of the target area of ​​the production line; The video stream information includes at least two frames of depth images of the target area; Determine, based on the depth data of the depth image and the depth data of an adjacent depth image, a fluctuation detection result for characterizing the fluctuation of the depth data; the adjacent depth image is a depth image adjacent to the depth image in image acquisition time; Based on the fluctuation detection result, it is determined whether there is an abnormality in the target area.

2. The method according to claim 1, wherein determining a fluctuation detection result for characterizing a fluctuation of the depth data based on the depth data of the depth image and the depth data of adjacent depth images comprises: Performing segmentation processing on each of the depth images to obtain at least one depth sub-image of each of the depth images; Different depth sub-images in each of the depth images have different image coding information, and in each of the depth images, depth sub-images at the same position of the depth image have the same image coding information; For each of the image coding information, based on the depth data of the depth image and the depth data of the adjacent depth image, determine whether there is at least one target depth sub-image corresponding to the image coding information in the at least two depth sub-images; the target depth sub-image is an image whose depth data fluctuation degree is greater than a preset value.

3. The method according to claim 2, wherein determining whether there is at least one target depth sub-image corresponding to the image encoding information in the at least two depth sub-images based on the depth data of the depth image and the depth data of the adjacent depth images comprises: Determine at least one depth sub-image pair in at least one depth sub-image corresponding to at least two frames of depth images respectively; The image coding information of different depth sub-images of each depth sub-image pair is the same, and the depth images corresponding to the different depth sub-images are adjacent in image acquisition time; Determining a change in depth data of the at least one depth sub-image pair; Based on the change conditions respectively corresponding to the at least one depth sub-image pair, it is determined whether there is at least one target depth sub-image corresponding to the image encoding information.

4. The method according to claim 3, for each of the depth sub-image pairs, determining a change in depth data of the at least one depth sub-image pair comprises: comparing depth data between two pixels of a pixel pair of the depth sub-image pair; The pixels in the pixel pair are at the same pixel position in the two depth sub-images of the depth sub-image pair; The number of pixel pairs whose depth data of the depth sub-image pair is changed is determined as the change of the depth data of the depth sub-image pair.

5. The method according to claim 4, wherein the number of pixel pairs having depth data changes comprises a first number of pixel pairs having depth data changes negatively, and a second number of pixel pairs having depth data changes positively; For each of the depth sub-image pairs, determining whether there is at least one target depth sub-image corresponding to the image encoding information based on the change conditions respectively corresponding to the at least one depth sub-image pair includes: When the first number and the second number are both greater than a first preset threshold, the depth sub-image with a later image acquisition time in the depth sub-image pair is determined as the target depth sub-image.

6. The method according to claim 5, wherein determining whether the target area has an abnormality based on the fluctuation detection result comprises: In a case where the target depth sub-image exists in the depth sub-images of the at least two frames of depth images, determining that an abnormality exists in the target area; In the case where the abnormality in the target area is a fire, the method further includes: Based on the target depth sub-image, determining the location information of the fire occurring in the target area and the flame area information; Determining a flame severity level of the fire based on the first number and the second number; The target depth sub-image, location information of a fire in the target area, flame area information, and flame severity level of the fire are output.

7. The method according to any one of claims 2 to 6, further comprising: Acquire at least one target depth sub-image corresponding to each image encoding information; Based on the image acquisition time, merging the target depth sub-images corresponding to the adjacent image encoding information to obtain a merged image; Based on the image coordinate information of each vertex of the merged image, determining the actual coordinate information of each vertex; Based on the actual coordinate information of each vertex, determining the actual area information corresponding to the depth data in the merged image whose fluctuation degree is greater than a preset value; The actual coordinate information of each vertex, the actual area information and the merged image are output.

8. The method according to any one of claims 2 to 6, further comprising: Performing differential processing on depth data of every two depth images that are adjacent in image acquisition time in the at least two frames of depth images; Depth images whose depth data have not changed as represented by the difference processing results are eliminated from the at least two depth image frames to obtain a processed depth image; the processed depth image is used to determine the fluctuation detection result.

9. An image processing device based on edge access, the image processing device comprising: An acquisition unit, used for acquiring video stream information of a target area of ​​a production line; The video stream information includes at least two frames of depth images of the target area; A first determining unit is used to determine a fluctuation detection result for characterizing the fluctuation of the depth data based on the depth data of the depth image and the depth data of an adjacent depth image; the adjacent depth image is a depth image adjacent to the depth image in image acquisition time; The second determining unit is used to determine whether there is an abnormality in the target area based on the fluctuation detection result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 8 are implemented.

Images