Steel bar stock change detection method, electronic equipment and storage medium
By using camera shooting guides in the steel bar warehouse to generate models and image change detection models, adjust the camera perspective and detect the changing areas of the steel bar stack, the accuracy and efficiency of steel bar inventory change detection are solved, and efficient in complex environments is achieved. Smart inventory supervision.
Patent Information
- Application Number
- CN202411001498.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-29
AI Technical Summary
The existing technology is difficult to accurately, efficiently and intelligently detect the inventory changes of rods and wires in reinforced warehouses. Especially in the case of irregular stacking and complex and changeable environment, the image shooting alignment quality and background pseudo-change noise have a great impact, resulting in Poor detection accuracy.
Using preset camera shooting guides to generate models and image change detection models, through real-time video stream acquisition and processing, adjust the camera shooting perspective, detect the image change area of the steel bar stack, calculate the initial and actual inventory changes, and perform inventory updates to ensure accurate detection. sex.
It realizes accurate and efficient detection of steel bar inventory changes in complex environments, reduces labor costs, improves the accuracy of counting results, avoids counting inaccurate problems caused by irregular stacking, and improves the actual availability of inventory change detection.
Smart Images

Figure CN119048798A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology and computer vision technology, and in particular to a steel bar inventory change detection method, electronic equipment and storage medium. Background Art
[0002] Rebar is a common and important building material and industrial material. It is widely used in the construction of buildings, urban infrastructure, etc., and its use demand is growing. Rebar can be mainly divided into bars represented by hot-rolled ribbed steel bars (commonly known as rebar), and wires represented by coils and high wires. Depending on the specific specifications and purpose of use, rebar is generally stored in indoor or outdoor rebar warehouses. The entry and exit of this type of rebar warehouse usually occurs very frequently, and the rebar warehouse inspectors need to regularly check the inventory of rebar every day. With the growing demand for the use of rebar, the accuracy of the rebar inventory and whether any rebar has been transferred to the wrong warehouse are crucial to rebar users such as construction teams.
[0003] At present, the supervision of steel bar inventory can be mainly carried out through two solutions: piece counting and change detection. Steel bar warehouse inspectors will adopt different solutions depending on the scenario, purpose and needs.
[0004] (1) Steel bar piece-counting refers to obtaining the specific number of steel bars in a steel bar pile by using pure manual counting, intelligent algorithm counting and other methods. However, in this solution, the manual counting method is time-consuming and labor-intensive, and the real-time performance is not high. The intelligent algorithm counting method can only detect the number of steel bars in the entire "tic-tac-toe" steel bar pile, and it is difficult to obtain the specific number of steel bars in each layer of such a steel bar pile, so it is impossible to know the specific inventory changes of the steel bar pile in detail. In addition, when the steel bar warehouse uses the "single-line" method to stack bars, few intelligent algorithms can accurately detect the specific inventory of such steel bar piles; and for wire rods, due to its special "offset and decreasing" stacking method, the steel bar piece-counting solution is difficult to accurately count its steel bar inventory. Therefore, the application scenarios targeted by this solution are limited, and it cannot achieve the purpose of flexible and lightweight supervision of steel bar inventory.
[0005] (2) Steel bar change detection refers to the use of video monitoring of steel bar pile status, manual comparison of steel bar pile appearance and other methods to directly detect whether the steel bar inventory has changed, and optionally perform quantitative analysis of inventory changes. In this scheme, the video monitoring method requires the deployment of several monitoring devices around each steel bar pile. However, since the operating environment of steel bar warehouses is generally very complex, it is usually forbidden to deploy too many fixed monitoring devices in the warehouse, so this method has poor universality. The manual comparison method introduces human subjective factors, and the accuracy of the inventory change judgment results is poor. In addition, manual comparison is time-consuming and labor-intensive, and the supervision efficiency is poor. In addition, considering the actual scene problems such as some steel bar piles in the warehouse occupying a large area and stacking irregularly, the steel bar warehouse environment is usually complex and changeable, and the shooting light brightness is uneven and difficult to control, and considering the existing artificial intelligence algorithms such as image change detection in the academic community, which are easily affected by the image detail alignment effect and the background pseudo-change noise, there are currently no application examples of artificial intelligence algorithms effectively applied to the steel bar inventory change detection task.
[0006] In summary, how to detect inventory changes of steel bars such as bars and wires more accurately, efficiently and intelligently in indoor or outdoor steel bar warehouses is a difficult problem that needs to be solved urgently. Summary of the invention
[0007] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a steel bar inventory change detection method, electronic device and storage medium, which can solve several technical challenges such as the accuracy of steel bar inventory change detection is easily affected by the image capture alignment quality and image background pseudo-change noise when several practical problems exist, such as irregular steel bar stacking, complex and changeable steel bar warehouse environment, and misaligned manually captured steel bar images. After using the steel bar inventory change detection method, electronic device and storage medium applied by the present invention, steel bar warehouse inspectors can accurately, efficiently and intelligently complete the steel bar inventory change detection task, and realize flexible and lightweight supervision of steel bar piles in complex, harsh and dangerous steel bar warehouses. The specific scheme proposed by the present invention is as follows:
[0008] In a first aspect, the present application provides a method for detecting steel bar inventory changes, comprising:
[0009] Acquire a real-time video stream of the steel bar pile obtained after real-time video stream collection of the steel bar pile;
[0010] A preset camera shooting guide generation model is used to detect a preset steel bar pile reference image and the current latest frame of the real-time video stream of the steel bar pile, and obtain camera shooting guides; the preset steel bar pile reference image is a historical state image of a specific shooting angle of the current specific steel bar pile, and the image usually has the ability to represent the inventory information of the steel bar pile;
[0011] Determine whether the camera shooting guide meets the preset camera shooting angle determination condition; if the camera shooting angle is suitable, temporarily store the current latest frame of the steel bar pile real-time video stream and regard it as the steel bar pile target image; if the camera shooting angle is not suitable, adjust the current shooting angle of the steel bar pile real-time video stream based on the camera shooting guide and using the preset steel bar pile real-time video stream acquisition control method; the steel bar pile target image is an image frame in the steel bar pile real-time video stream that is close to the preset steel bar pile reference image shooting angle;
[0012] The preset image change detection model and the preset target detection model are used to detect the preset steel bar pile reference image and the steel bar pile target image, and the steel bar pile image change area of the steel bar pile target image compared with the preset steel bar pile reference image, and the initial change inventory number of steel bars such as rods or wires are obtained; the steel bar pile image change area is a pixel area in the input steel bar pile image that has semantic changes such as addition or deletion, and has the ability to characterize the steel bar inventory change between the current moment and a certain historical moment; the initial change inventory number is the actual number of steel bars detected by the target detection model that have inventory changes such as addition or deletion in the steel bar pile image change area;
[0013] Determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number. If not, update the initial change inventory number through a preset steel bar inventory update method to obtain a target change inventory number that is consistent with the actual change inventory number.
[0014] Optionally, the step of acquiring a real-time video stream of the steel bar pile obtained by collecting the real-time video stream of the steel bar pile includes:
[0015] Use a mobile phone or other camera device with image and video acquisition functions to collect video of the steel bar pile in real time to obtain the original video stream from the camera;
[0016] By presetting the original video stream post-processing method, the original video stream from the camera is converted into a real-time video stream of the steel bar pile that meets the processing format of the subsequent steps.
[0017] Optionally, the step of obtaining the camera shooting guide by using a preset camera shooting guide generation model includes:
[0018] Select one of the camera shooting guide generation models with better effect in the corresponding steel bar pile shooting scene from the preset camera shooting guide generation model set;
[0019] The first sub-step of generating the model by using the camera shooting guide and inputting two images of the steel bar pile taken before and after to obtain the absolute position change of the camera between the two shots, that is, the relative position of the camera;
[0020] The second sub-step of using the camera shooting guide to generate a model is to post-process the relative position of the camera into a camera shooting guide to prompt the operator how to adjust the shooting angle of the camera.
[0021] Optionally, a camera shooting guide generation model is constructed using a preset camera motion estimation model based on artificial intelligence and computer vision, including:
[0022] Using a pre-collected camera motion estimation training set to train an initial model built based on artificial intelligence and computer vision algorithms to obtain the preset camera motion estimation model;
[0023] By utilizing the preset camera motion estimation model and encapsulating the steel bar pile image detection logic and retaining a number of intermediate detection results, a camera shooting guide generation model is finally constructed in the form of two sub-steps.
[0024] Optionally, the two sub-steps of using a preset camera shooting guideline to generate a model to obtain a camera shooting guideline include:
[0025] In the first sub-step, a preset camera motion estimation algorithm for constructing the camera shooting guide generation model is used to detect the two images taken before and after, and obtain the relative position and posture of the camera before and after shooting;
[0026] In a second sub-step, a preset camera posture post-processing method is used to analyze and process the detected camera relative posture to generate a camera shooting guide that is in an inverse relationship with the camera relative posture.
[0027] Optionally, the determining whether the camera shooting angle is appropriate includes:
[0028] According to the characteristics of the camera shooting guide generation model, a method of calculating and comparing preset thresholds such as posture physical distance is used to determine whether the current camera's shooting angle of view is similar to the historical angle of view.
[0029] Optionally, the method of adjusting the current shooting angle of the real-time video stream of the steel bar pile by using a preset steel bar pile real-time video stream acquisition and control mode includes:
[0030] Based on the camera shooting guide, a text reminder program and / or a voice reminder program is executed to perform a prompt operation for obtaining the steel bar pile image with a suitable viewing angle, so as to obtain the steel bar pile image with a shooting viewing angle closer according to the prompt operation.
[0031] Optionally, a preset image change detection model is used to detect the change area of the steel bar pile image, including:
[0032] The initial model constructed based on the artificial intelligence algorithm is trained using a pre-made image change detection training set to obtain the preset image change detection model for detecting the change area of the steel bar pile image.
[0033] Optionally, the calculating the initial change inventory number of steel bars such as bars or wires by using a preset target detection model includes:
[0034] The initial model constructed based on the artificial intelligence algorithm is trained using a pre-made target detection training set to obtain the preset target detection model.
[0035] Optionally, the determining whether the actual change in inventory quantity is consistent with the initial change in inventory quantity includes:
[0036] After obtaining the initial change inventory number through the preset image change detection model and the preset target detection model, the preset steel bar inventory consistency judgment method is used to determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number. If they are inconsistent, the preset steel bar inventory update method is used to update the initial change inventory number.
[0037] In a second aspect, the present application provides a device for detecting changes in steel bar inventory, comprising:
[0038] A steel bar pile real-time video stream acquisition module is used to acquire the steel bar pile real-time video stream obtained after real-time video stream acquisition of the steel bar pile;
[0039] A camera shooting guide generation module is used to detect a preset steel bar pile reference image and the current latest frame of the real-time video stream of the steel bar pile using a preset camera shooting guide generation model, and obtain camera shooting guides; the preset steel bar pile reference image is a historical state image of a specific shooting angle of the current specific steel bar pile, and the image usually has the ability to represent the inventory information of the steel bar pile;
[0040] A steel bar pile target image acquisition module is used to determine whether the camera shooting guide meets the preset camera shooting angle determination condition. If the camera shooting angle is suitable, the current latest frame of the steel bar pile real-time video stream is temporarily stored and regarded as the steel bar pile target image. If the camera shooting angle is not suitable, the current shooting angle of the steel bar pile real-time video stream is adjusted based on the camera shooting guide and using the preset steel bar pile real-time video stream acquisition and control method; the steel bar pile target image is an image frame in the steel bar pile real-time video stream that is close to the preset steel bar pile reference image shooting angle;
[0041] An initial change inventory number acquisition module is used to detect the preset steel bar pile reference image and the steel bar pile target image using a preset image change detection model and a preset target detection model to obtain a steel bar pile image change area of the steel bar pile target image compared to the preset steel bar pile reference image, and an initial change inventory number of steel bars such as rods or wires; the steel bar pile image change area is a pixel area in the input steel bar pile image where semantic changes such as addition or deletion exist, and has the ability to characterize the steel bar inventory change between the current moment and a certain historical moment; the initial change inventory number is the actual number of steel bars detected by the target detection model where inventory changes such as addition or deletion exist in the steel bar pile image change area;
[0042] The target change inventory number acquisition module is used to determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number. If they are inconsistent, the initial change inventory number is updated through a preset steel bar inventory update method to obtain a target change inventory number that is consistent with the actual change inventory number.
[0043] In a third aspect, the present application provides an electronic device for detecting changes in steel bar inventory, comprising:
[0044] Memory, used to store computer programs;
[0045] The processor is used to execute the computer program to implement the above-mentioned steel bar inventory change detection method.
[0046] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned steel bar inventory change detection method when executed by a processor.
[0047] In the present invention, a real-time video stream of a steel bar pile is obtained after real-time video stream collection of the steel bar pile; a preset camera shooting guide generation model is used to detect a preset steel bar pile reference image and the current latest frame of the real-time video stream of the steel bar pile, and a camera shooting guide is obtained; it is judged whether the camera shooting guide meets the preset camera shooting angle determination condition, if the camera shooting angle is suitable, the current latest frame of the real-time video stream of the steel bar pile is temporarily stored and regarded as a target image of the steel bar pile; if the camera shooting angle is not suitable, the current shooting angle of the real-time video stream of the steel bar pile is adjusted based on the camera shooting guide and using the preset real-time video stream collection control mode of the steel bar pile; the preset image change detection model and the preset target detection model are used to detect the preset steel bar pile reference image and the target image of the steel bar pile, and the steel bar pile image change area of the target image of the steel bar pile compared with the preset steel bar pile reference image, and the initial change inventory number of steel bars such as bars or wires are obtained; it is judged whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number, if not, the initial change inventory number is updated by a preset steel bar inventory update method to obtain a target change inventory number consistent with the actual change inventory number. That is, the present application utilizes a preset camera shooting guide to generate a model to detect and manipulate the real-time video stream of the steel bar pile to obtain a target steel bar pile image that is close to the perspective of the steel bar pile reference image, and then utilizes a preset image change detection model and a preset target detection model to detect the initial change inventory results of the steel bar inventory change in the reference and target historical steel bar pile images, and then determines whether the actual change inventory number is consistent with the initial change inventory number to perform inventory updates and obtain the target change inventory number. It should be pointed out that the steel bar inventory change detection proposed by the present invention that relies on the above steps and their optional steps is a solution that has not been proposed or involved in existing systems and studies, and has certain innovation and practical significance.
[0048] Compared with the prior art, the present application has the following advantages and beneficial effects: the shooting angle of the real-time video stream of the steel bar pile is controlled by the model generated according to the preset camera shooting guide, and the shooting angle of the steel bar pile can be intelligently and efficiently aligned when the problem of misalignment of the steel bar image is present in the manual shooting; the image change area is pre-detected and obtained according to the preset image change detection model, and the detection accuracy of the steel bar pile image change area can be improved when the steel bar warehouse environment is complex and changeable, and there is image background pseudo-change noise, etc., and by cooperating with the subsequent model, the steel bar inventory change detection capability can be obtained, and the manual comparison of steel bar pile changes can be avoided, and the labor cost and time cost can be reduced; the initial inventory change is detected and counted according to the preset target detection model, and the manual counting of steel bar change inventory can be avoided, and the labor cost and time cost can be reduced, and the accuracy of the counting result can be improved when the steel bars are irregularly stacked and the large-scale densely stacked steel bars are counted; the inventory is updated according to whether the actual change inventory number is consistent with the initial change inventory number, so as to prevent the inaccurate steel bar counting problem caused by the large-scale dense stacking of steel bars from affecting the inventory change detection, and the accuracy of the steel bar inventory change detection is further improved by modifying the erroneous counting result, which is convenient for user operation and improves the actual usability of the steel bar inventory change detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0050] Figure 1 A flow chart of a steel bar inventory change detection method provided in this application;
[0051] Figure 2 A specific flow chart of a steel bar inventory change detection method provided for this application;
[0052] Figure 3 A schematic diagram of a steel bar sequence image dataset provided for this application;
[0053] Figure 4 A schematic diagram of the result of estimating camera motion for a steel bar sequence provided in this application;
[0054] Figure 5 A schematic diagram of a steel bar image change detection dataset provided in this application;
[0055] Figure 6 A schematic diagram of the result of detecting image changes of a steel bar pile image provided by the present application;
[0056] Figure 7 A schematic diagram of a steel bar bundle target detection dataset provided for this application;
[0057] Figure 8 A schematic diagram of the actual operation process of a mobile terminal provided for this application;
[0058] Fig. 9 A schematic diagram of the actual operation process of a mobile terminal provided for this application;
[0059] Fig.10 A schematic diagram of the actual operation process of a mobile terminal provided for this application;
[0060] Fig.11 A schematic diagram of the structure of a steel bar inventory change detection device provided in this application;
[0061] Fig.12 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION
[0062] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0063] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0064] In the description of the present invention, the meaning of "several" is one or more, the meaning of "more" is two or more, and the meanings of "greater than", "less than", "exceed" and the like are understood as not including the number itself, and the meanings of "above", "below", "within" and the like are understood as including the number itself. If there is a description of the first and the second, it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, "and / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.
[0065] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0066] At present, how to detect the inventory changes of steel bars such as bars and wires more accurately, efficiently and intelligently in indoor or outdoor steel warehouses is a difficult problem to be solved urgently. Therefore, the present invention discloses a steel bar inventory change detection method, which can solve several technical challenges such as the accuracy of steel bar inventory change detection is easily affected by the image shooting alignment quality and image background pseudo-change noise when several practical problems exist, such as irregular steel bar stacking, complex and changeable steel bar warehouse environment, and misaligned steel bar images taken manually. After using the steel bar inventory change detection method, device, equipment and storage medium applied by the present invention, steel bar warehouse inspectors can accurately, efficiently and intelligently complete the steel bar inventory change detection task, and realize flexible and lightweight supervision of steel bar piles in complex, harsh and dangerous steel bar warehouses.
[0067] Example 1
[0068] See also Figure 1 As shown, an embodiment of the present invention discloses a method for detecting steel bar inventory changes, comprising:
[0069] Step S11, obtaining a real-time video stream of the steel bar pile obtained after real-time video stream collection of the steel bar pile.
[0070] In this embodiment, it should be pointed out that the real-time video stream of the steel bar pile obtained after the real-time video stream of the steel bar pile is acquired includes: using a camera device with image and video acquisition functions such as a mobile phone to perform real-time video acquisition on the steel bar pile to obtain the original video stream from the camera; by presetting the original video stream post-processing method, the original video stream from the camera is converted into a real-time video stream of the steel bar pile that conforms to the processing format of the subsequent steps. For example, the camera on the mobile phone can be used to take real-time photos of the steel bar pile in the steel bar warehouse, and optionally perform image color space conversion processing to obtain the real-time video stream of the steel bar pile.
[0071] Step S12, using a preset camera shooting guidance generation model to detect a preset steel bar pile reference image and the current latest frame of the real-time video stream of the steel bar pile, and obtain camera shooting guidance; the preset steel bar pile reference image is a historical state image of a specific shooting angle of the current specific steel bar pile, and the image usually has the ability to characterize the inventory information of the steel bar pile.
[0072] In this embodiment, it should be pointed out that the use of the preset camera shooting guide generation model to obtain the camera shooting guide includes: selecting one of the camera shooting guide generation models with better effect in the corresponding steel bar pile shooting scene from the preset camera shooting guide generation model set; using the first sub-step of the camera shooting guide generation model and inputting two steel bar pile images taken before and after to obtain the absolute camera posture change of the two shots, that is, the relative camera posture; using the second sub-step of the camera shooting guide generation model to post-process the relative camera posture into a camera shooting guide to prompt the operator how to adjust the camera's shooting angle. That is, the camera shooting guide generation model is one of the models selected from the preset camera shooting guide generation model set that is most suitable for the current steel bar pile shooting scene, forward calculation is performed on the historically shot steel bar pile image and the steel bar pile image extracted from the real-time video stream, and the camera shooting guide is obtained by using the camera posture post-processing method after obtaining the relative camera posture of the two shots. Among them, the camera shooting guide generation model set is a model set formed by constructing several camera shooting guide generation models based on traditional computer vision algorithms such as feature points and three-dimensional camera sets and / or artificial intelligence and computer vision algorithms. The former is more targeted at shooting scenes with obvious feature textures in the captured images, while the latter is more targeted at more general shooting scenes such as no obvious feature textures in the captured images; the basic model used to construct the camera shooting guide generation model is usually a camera motion estimation model, and the camera shooting guide generation model with better effect needs to be selected after evaluating the characteristics of different steel bar pile shooting scenes. It can be understood that the camera motion estimation model used to construct the camera shooting guide generation model and constitute the camera shooting guide generation model set in this embodiment is not limited to the above examples. Any model or similar model that can be used to estimate camera motion can serve as the basic model for constructing the camera shooting guide generation model. In this way, the prediction and generation effect of the camera shooting guide in the steel bar warehouse can be maximized, the efficiency and accuracy of the steel bar pile video stream acquisition can be further improved, and it is helpful to improve the accuracy of subsequent image change detection, so as to obtain more accurate steel bar inventory change detection results.
[0073] It should be noted that the use of a preset camera motion estimation model based on artificial intelligence and computer vision to construct a camera shooting guidance generation model includes: using a pre-collected camera motion estimation training set to train an initial model constructed based on an artificial intelligence and computer vision algorithm to obtain the preset camera motion estimation model; using the preset camera motion estimation model and encapsulating the steel bar pile image detection logic and retaining a number of intermediate detection results, and finally constructing a camera shooting guidance generation model in the form of two sub-steps. The camera motion estimation training set includes steel bar pile images including bars and wires, and optional annotation information that characterizes the camera shooting perspective characteristics.
[0074] It should be pointed out that the two sub-steps of using the preset camera shooting guide generation model to obtain the camera shooting guide include: a first sub-step, using the preset camera motion estimation algorithm for constructing the camera shooting guide generation model to detect the two images taken before and after, and obtain the relative camera pose before and after shooting; a second sub-step, using the preset camera pose post-processing method to analyze and process the detected camera relative pose to generate a camera shooting guide that is in an inverse relationship with the camera relative pose. That is, the generation of the camera shooting guide needs to first rely on the preset camera motion estimation algorithm to detect images at two different historical moments and obtain the camera relative pose, and then analyze and process the camera relative pose to obtain the correct camera shooting guide that is in an inverse relationship with the camera relative pose. Among them, the relative pose of the camera includes two components, rotation and translation, and some camera motion estimation algorithms can detect the pose rotation component and pose translation component with actual physical meaning, but other camera motion estimation algorithms can only detect the pose rotation component with actual physical meaning, and cannot detect the pose translation component with actual physical meaning due to the scale uncertainty problem. That is, through some camera motion estimation algorithms, this embodiment may obtain the camera relative pose result of "the camera optical center translation (1 meter, 0 meter, -1 meter) and the phase plane rotation (-30 degrees, 45 degrees, -60 degrees) in the world coordinate system"; through other camera motion estimation algorithms, this embodiment may obtain the camera relative pose result of "the camera optical center translation (1, 0, -1) and the phase plane rotation (-30 degrees, 45 degrees, -60 degrees) in the world coordinate system". It can be understood that in the example of the second camera motion estimation algorithm, the pose translation component does not carry any unit with physical meaning, which means that such pose translation component only has physical meaning in the translation direction. In addition, the inverse relationship is that the detected camera relative posture result and the generated camera shooting guidance result have the characteristics of reverse translation and reverse rotation, thereby achieving the purpose of compensating for the camera movement to adjust the current camera shooting angle to the historical shooting angle. The camera shooting guidance, for example, "the camera moves 1 meter to the right and 1 meter down" or "the camera moves 45 degrees to the right and down", will be used in subsequent steps.
[0075] Step S13, determine whether the camera shooting guide meets the preset camera shooting angle determination conditions; if the camera shooting angle is suitable, temporarily store the current latest frame of the real-time video stream of the steel bar pile and regard it as the target image of the steel bar pile; if the camera shooting angle is not suitable, adjust the current shooting angle of the real-time video stream of the steel bar pile based on the camera shooting guide and using the preset real-time video stream acquisition control method of the steel bar pile; the target image of the steel bar pile is an image frame in the real-time video stream of the steel bar pile that is close to the shooting angle of the preset steel bar pile reference image.
[0076] In this embodiment, it should be pointed out that the determination of whether the camera shooting angle is appropriate includes: according to the characteristics of the camera shooting guidance generation model adopted, the method of calculating and comparing preset thresholds such as the physical distance of the posture is used to determine whether the shooting angle of the current camera is close to the historical angle. That is, after obtaining the camera shooting guidance, for different steel bar pile shooting scenes, according to the intermediate detection results and the final detection results of the specific camera shooting guidance generation model, the preset camera shooting angle determination conditions such as threshold judgment are used to determine whether the shooting angles of the two images are close enough. It can be understood that the determination condition of whether the camera shooting angle is appropriate or not in this embodiment is not limited to the judgment of the physical distance threshold of the posture. Any method that uses the intermediate detection results and the final detection results of the camera shooting guidance generation model or the steel bar pile image taken by the camera and has the ability to determine whether the camera shooting angle is close when shooting the steel bar pile image twice can serve as the determination condition of whether the camera shooting angle is appropriate or not.
[0077] It should be pointed out that the use of a preset steel bar pile real-time video stream acquisition control method to adjust the current shooting angle of view of the steel bar pile real-time video stream includes: executing a text reminder program and / or a voice reminder program based on the camera shooting guide to perform a prompt operation for obtaining a steel bar pile image with a suitable angle of view, so as to obtain a steel bar pile image with a shooting angle closer to the prompt operation. That is, when it is known that the current camera shooting angle of view is not close enough to the preset steel bar pile reference image shooting angle of view based on the judgment condition of whether the shooting angle of view is suitable or not, a text reminder program and / or a voice reminder program are executed based on the camera shooting guide to perform a prompt operation for obtaining a steel bar pile image with a suitable angle of view. Among them, the text reminder program and the voice reminder program can be an electronic device with relevant functions using the present invention, and the execution of the text reminder program and / or the voice reminder program can be determined according to user needs.
[0078] Step S14, using a preset image change detection model and a preset target detection model to detect the preset rebar pile reference image and the rebar pile target image, to obtain the rebar pile image change area of the rebar pile target image compared with the preset rebar pile reference image, and the initial change inventory number of rebars such as bars or wires; the rebar pile image change area is a pixel area in the input rebar pile image where semantic changes such as addition or deletion exist, and has the ability to characterize the rebar inventory change between the current moment and a certain historical moment; the initial change inventory number is the actual number of rebar pieces detected by the target detection model where inventory changes such as addition or deletion exist in the rebar pile image change area.
[0079] In this embodiment, it should be pointed out that the detection of the rebar pile image change area using the preset image change detection model includes: using a pre-made image change detection training set to train the initial model constructed based on the artificial intelligence algorithm to obtain the preset image change detection model for detecting the rebar pile image change area. Among them, the image change detection training set includes a pair of rebar pile images including bars and wires, and image change area annotation information characterizing the image change area in the image pair. That is, before detecting the change in the rebar inventory, the preset image change detection model is used to compare two images of the same rebar pile to be detected at different times and detect the rebar pile image change area. In this way, when the preset target detection model is used to detect the total rebar inventory in the future, the rebar change inventory can be detected simultaneously by relying on the rebar pile image change area.
[0080] It should be pointed out that the use of a preset target detection model to calculate the initial change inventory number of steel bars such as bars or wires includes: using a pre-made target detection training set to train the initial model constructed based on an artificial intelligence algorithm to obtain the preset target detection model. The target detection training set includes an image of a steel bar pile containing bars and wires, and annotation information that characterizes the type of steel bar target features contained in the image. In addition, the preset target detection model can forward calculate the change area of the steel bar pile image and calculate the steel bar inventory number in the change area, that is, the steel bar change inventory of the steel bar pile, that is, the initial change inventory number; the preset target detection model can also perform forward calculation on the steel bar pile target image and calculate the steel bar inventory number in the image, and then filter the change area of the steel bar pile image to obtain the steel bar change inventory of the steel bar pile, that is, the initial change inventory number.
[0081] Step S15, determining whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number; if not, updating the initial change inventory number through a preset steel bar inventory update method to obtain a target change inventory number that is consistent with the actual change inventory number.
[0082] In this embodiment, it should be pointed out that the determination of whether the actual change inventory number is consistent with the initial change inventory number includes: after obtaining the initial change inventory number through the preset image change detection model and the preset target detection model, the preset steel bar inventory consistency determination method is used to determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number, and if not, the preset steel bar inventory update method is used to update the initial change inventory number. Wherein, the actual change inventory number is the number of steel bars that actually undergo inventory changes in the specific steel bar pile; the preset steel bar inventory consistency determination method is usually an operator manually detecting the steel bar inventory change detection system deployed in the present invention and determining whether there are steel bars that are missed in the system output result; the steel bar inventory update method is usually an operator operating the steel bar inventory change detection system deployed in the present invention and interacting through the interface to update the steel bar change inventory number displayed in the system. In this way, the method can further improve the accuracy of steel bar inventory change detection, facilitate user operation, and improve the actual usability of steel bar inventory change detection.
[0083] In this embodiment, the shooting angle of the real-time video stream of the steel bar pile is manipulated by the model generated according to the preset camera shooting guidance, so that the shooting angle of the steel bar pile can be intelligently and efficiently aligned when the problem of misalignment of the steel bar image is present in the manual shooting; the image change area is pre-detected and obtained according to the preset image change detection model, so that the detection accuracy of the steel bar pile image change area can be improved when the steel bar warehouse environment is complex and changeable and there is image background pseudo-change noise, and by cooperating with the subsequent model, the steel bar inventory change detection capability can be obtained, avoiding manual comparison of steel bar pile changes, reducing labor costs and time costs; the initial inventory changes are detected and counted according to the preset target detection model, so that manual counting of steel bar change inventory can be avoided, reducing labor costs and time costs, and the accuracy of the counting results can be improved when the steel bars are irregularly stacked and large-scale densely stacked steel bars are counted; the inventory is updated according to whether the actual change inventory number is consistent with the initial change inventory number, so as to prevent the inaccurate steel bar counting problem caused by large-scale dense stacking of steel bars from affecting the inventory change detection, and the accuracy of the steel bar inventory change detection is further improved by modifying the erroneous counting results, which is convenient for user operation and improves the actual usability of the steel bar inventory change detection. The invention proposes to detect steel bar inventory changes by relying on the above steps and optional steps, which is a solution not proposed or involved in existing systems and studies, and has innovation and practical significance.
[0084] Example 2
[0085] See also Figure 2 As shown, the embodiment of the present invention discloses a specific method for detecting steel bar inventory changes, including:
[0086] Step S21, obtaining a target image of the steel bar pile after real-time video stream capture of the steel bar pile at a historical moment and obtaining the target image.
[0087] In this embodiment, it should be pointed out that the acquisition of the target image of the rebar pile acquired at a historical moment includes: acquiring a target image of the rebar pile with a shooting angle close to that of the reference image of the rebar pile through steps S11 to S13 of embodiment 1 before the historical moment; and then acquiring the target image of the rebar pile acquired at any subsequent moment for detection of changes in rebar inventory. That is, the target image of the rebar pile described in this embodiment is an image acquired and obtained by an operator at a specific angle of view of a specific rebar pile at a historical moment. It is not necessary to acquire the real-time video stream of the rebar pile again and rely on a preset camera shooting guide generation algorithm to obtain the target image of the rebar pile when the same multiple rebar inventory change detections are required. Instead, it is allowed to directly select the target image of the rebar pile for subsequent steps to detect changes in rebar inventory. In this way, the user experience can be greatly improved, and the purpose of more flexible detection can be achieved.
[0088] Step S22, using a preset image change detection model and a preset target detection model to detect the preset rebar pile reference image and the rebar pile target image, to obtain the rebar pile image change area of the rebar pile target image compared with the preset rebar pile reference image, and the initial change inventory number of rebars such as bars and wires; the rebar pile image change area is a pixel area in the input rebar pile image where semantic changes such as addition or deletion exist, and has the ability to characterize the rebar inventory change between the current moment and a certain historical moment; the initial change inventory number is the actual number of rebar pieces detected by the target detection model where inventory changes such as addition or deletion exist in the rebar pile image change area.
[0089] Step S23, determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number. If they are inconsistent, update the initial change inventory number through a preset steel bar inventory update method to obtain a target change inventory number that is consistent with the actual change inventory number.
[0090] For the specific process of step S22 and step S23, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.
[0091] In this embodiment, by directly acquiring the image of the steel bar pile obtained after image acquisition of the steel bar pile, after the target image of the steel bar pile has been acquired at a historical moment, there is no need to acquire the real-time video stream of the steel bar pile again and rely on a preset camera shooting guide generation algorithm to obtain the target image of the steel bar pile. Instead, it is allowed to detect steel bar inventory changes by directly selecting the target image of the steel bar pile, which can greatly improve the user experience and achieve the purpose of more flexible detection. The advantages and beneficial effects of the steel bar inventory change detection steps other than these can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here. The steel bar inventory change detection proposed by the present invention that relies on the above steps and their optional steps is a solution that has not been proposed or involved in existing systems and studies, and has certain innovation and practical significance.
[0092] Example 3
[0093] A possible implementation of the present invention is described in detail below in conjunction with the steel bar inventory change detection process disclosed in the present invention.
[0094] In order to achieve more accurate, efficient and intelligent inventory change detection of steel bars such as bars and wires in indoor or outdoor steel warehouses, the present invention discloses a steel bar inventory change detection process. In order to solve several practical problems such as irregular steel bar stacking, complex and changeable steel bar warehouse environment, and misalignment of manually photographed steel bar images, as well as several technical challenges such as the accuracy of steel bar inventory change detection is easily affected by image shooting alignment quality and image background pseudo-change noise, the present invention introduces a camera shooting guidance generation model, an image change detection model, and a target detection model that are suitable for the scene and can solve the corresponding problems in the disclosed steel bar inventory change detection process. In order to obtain a high-quality model in the steel bar warehouse scene where the above-mentioned problems and challenges exist and realize steel bar inventory change detection that meets user needs, you can go to places such as steel bar warehouses and use mobile phones and other camera equipment with image acquisition functions and video acquisition functions to collect steel bar pile images and steel bar pile videos of bars and wires, and it is necessary to cover the above-described scene characteristics and the steel bar pile status at different historical moments as much as possible, so that the trained camera shooting guidance generation model, image change detection model, and target detection model have higher accuracy and better generalization.
[0095] For the camera shooting guidance generation model, this embodiment constructs a steel bar sequence image dataset for training the camera motion estimation model from the several steel bar pile videos collected above through video filtering, resizing, image cropping, video frame extraction, brightness adjustment and other operations, and is used to construct a camera shooting guidance generation model based on artificial intelligence and computer vision algorithms for more general shooting scenarios. It should be mentioned that this embodiment adopts the idea of unsupervised learning to construct the camera motion estimation model, so it does not require the use of camera equipment with sensors such as odometers and depth cameras, nor does it require any annotation of the dataset. After data division, the training set part of the steel bar sequence image dataset contains 14 video sequences with a total of 1564 images, and the verification set part contains 4 video sequences with a total of 384 images, and their resolutions are all processed to 640x384. As an example, Figure 3 Shows some images of the steel bar sequence image dataset.
[0096] This embodiment uses depth estimation as an auxiliary task, together with the pose estimation task, to construct a camera motion estimation model, which consists of a depth estimation network and a pose estimation network. Similar to related research works such as SfMLearner and SC-Depth, the depth estimation network outputs two dense depth maps by inputting two images at different times, and the pose estimation network outputs the relative pose of the camera motion in this process by inputting two images at different times, and then generates the corresponding camera shooting instructions according to step S12 of embodiment 1. Considering that in order to increase the generation speed of the camera shooting instructions and reduce the impact of the network transmission rate between the mobile terminal used for real-time video stream collection of steel bar piles and the model inference server, it is necessary to deploy the above model to the mobile terminal for operation. Therefore, in the above-constructed model, this embodiment innovatively uses MobileNetV3 as the feature encoding layer of the pose estimation network in the pose estimation network. The structure uses lightweight depth separable convolution, has extremely fast reasoning speed, and can achieve relatively good feature encoding by cooperating with a carefully designed convolution layer structure. In this way, when performing camera motion estimation, there is no need to use ordinary convolutions with complicated calculations, but only deep separable convolutions are needed to quickly obtain deep abstract features of the input image, and fast forward reasoning is achieved on mobile devices such as mobile phones with optional neural acceleration units, and the camera shooting guidance generation model with fast reasoning can be achieved by converting the model to ONNX format for deployment. In addition, in order to further improve the accuracy of camera motion estimation, this embodiment introduces an internal point mask calculation module based on feature map broadcast multiplication feature correlation calculation for the steel warehouse scene with complex and changeable environment, and applies it before the pose decoding layer of the pose estimation network. By spontaneously fitting the output of the module to the self-discovery mask calculated based on the two depth maps predicted by the depth estimation network, the pixel point area with similar and matching features in the input image pair is explicitly extracted, so that the pose estimation network pays more attention to such areas when extracting image feature maps, reducing the influence and interference of shadow interference, dynamic objects, scene changes, etc. on motion estimation during front and back shooting, and further improving the camera shooting guidance generation capability for the steel bar pile video stream in the steel bar warehouse application scenario.
[0097] For the camera motion estimation model used to realize the generation of camera shooting guidance in this embodiment, the steel bar sequence image dataset is used for model training, and the total iteration rounds are set to 100, the single batch size is 6, the initial learning rate of training is 0.0001, and the learning rate exponential prediction and learning rate cosine decay strategy are adopted. The loss function in this embodiment consists of four parts, and the specific formula is:
[0098] Loss = λ photometric loss photometric +λ smooth losssmooth +λ geometric loss geometric +λ mask loss mask
[0099] Among them, Loss is the total loss, loss photometric The loss is the photometric loss calculated based on the pixel difference of the input image and the structural similarity algorithm SSIM. smooth is the smoothness loss calculated based on the gradient of the input image and the predicted depth, loss geometric The geometric consistency loss is calculated based on the difference and ratio of the two predicted depth maps. mask The inlier mask loss innovatively proposed by this embodiment is used for fitting the inlier mask. Since the camera motion estimation model used to construct the camera shooting guide generation model disclosed in this embodiment adopts the idea of unsupervised learning, its fitting difficulty is greater than that of the supervised learning model. Therefore, in order to weigh the weights between different loss terms, this embodiment also discloses a pair of loss weights with a certain fitting effect, namely λ photometric , smooth , geometric , mask are 1.0, 0.1, 0.5, and 0.05 respectively. This embodiment uses the PyTorch deep learning computing framework to train the model in the steel bar sequence image dataset training set. Based on the camera motion estimation model described in this embodiment, Figure 4 As shown in the figure, randomly selecting a steel bar pile video sequence in the corresponding validation set can obtain good camera motion estimation results.
[0100] It should be pointed out that the method for judging whether the camera shooting angle is appropriate or not in step S13 of embodiment 1 includes: for the steel bar pile shooting scene and the camera shooting guide generation model without scale problem, this embodiment can judge whether the shooting angle is appropriate or not by the way that the physical distance of the posture is not greater than the preset threshold; and for the steel bar pile shooting scene and the camera shooting guide generation model with scale problem, this embodiment can judge whether the shooting angle is appropriate or not by the way that the posture rotation component is not greater than the preset threshold and other specific preset conditions. For the camera motion estimation model based on artificial intelligence described in this embodiment, the latter specific preset condition can be the image similarity threshold calculated based on the internal point mask and the structural similarity algorithm SSIM, such as the following formula. However, it can be understood that the camera shooting angle determination condition described in this embodiment is not limited to the above example. Any method that can match the intermediate detection result and the final detection result of the camera shooting guide generation model and can judge whether the camera shooting angle is close when shooting the steel bar pile image twice can serve as the camera shooting angle determination condition.
[0101] ||r1-r2||2≤γ r
[0102] [1-SSIM(I s ×M i ,I t ×M i )] / (2×||M i ||1)≤γ s
[0103] For the image change detection model, this embodiment constructs a data set for training the image change detection model from the several steel bar pile images collected above, also through image filtering, resizing, image cropping, brightness adjustment and other operations. Specifically, this embodiment organizes the steel bar pile images of different stacking positions and different perspectives taken at different times, obtains several pairs of images of the same steel bar pile at different times but with similar perspectives through grouping and permutation, and uses the LabelMe tool to annotate the steel bar foreground area in the image pair and the steel bar area with inventory changes, to construct a steel bar image change detection data set for model training. After data partitioning, the training set part of the data set contains a total of 188 pairs of images, and the validation set part contains a total of 62 pairs of images, and their resolutions are all processed to 640x480. This embodiment also uses a number of randomly generated affine transformations, thin plate transformations, and homography transformations to expand the data set to include 750 pairs of images in the training set and 250 pairs of images in the validation set. In addition to the semantic segmentation binary mask labels and the change region binary mask labels, some of the data generated by image transformation also has optical flow labels that can be used for supervised training. As an example, Figure 5 Shows some images and their labels of the rebar image change detection dataset.
[0104] This embodiment uses optical flow estimation and semantic segmentation as auxiliary tasks, together with the change detection task, to build an image change detection model. The model consists of a common feature encoding module, as well as a task-related optical flow decoding module, a segmentation decoding module, and a change decoding module, a total of four parts. Similar to related research works such as SimSaC and GLU-Net, the feature encoding module is mainly composed of a feature pyramid, and this module is applied at high and low resolutions through downsampling processing to encode image depth abstract features of different scales. In order to achieve the two tasks of optical flow estimation and change detection, this embodiment also refers to existing research work, introduces global correlation based on global broadcast multiplication of feature maps and local correlation based on broadcast multiplication in a small area of feature maps, which are used to describe the relationship between different pixels of two feature maps, including the correlation representing optical flow and the irrelevance representing change. In order to better extract the deep features of steel bar images in a steel bar warehouse with a complex environment, this embodiment innovatively introduces a feature refinement layer in the feature encoding module to improve the quality of feature encoding. Considering that the multi-task learning task of change detection is relatively complex, this embodiment uses the VGG16 network pre-trained based on the ImageNet image classification data set to construct a multi-scale feature pyramid. In order to improve the transfer learning effect, this embodiment includes multiple layers of 2D convolution and improved CBAM attention in the introduced feature refinement layer. Specifically, this embodiment replaces the single 7x7 convolution in the original CBAM spatial attention with a double-layer 7x7 convolution to cooperate with the downsampling scale and expand the receptive field, thereby realizing the capture of the global attention information of the feature map and improving the feature encoding effect.
[0105] Different tasks in the image change detection model constructed in this embodiment use specific feature fusion methods to make use of the domain knowledge of different supervisory signals, and finally form a complex data flow structure. Specifically, in order to solve the problem that the accuracy of steel bar inventory change detection is easily affected by the quality of image alignment, the model introduces optical flow estimation as an auxiliary task, and uses FlowNet as its decoding layer structure. By inputting the feature correlation and the image depth abstract features at different scales obtained by the feature encoding module, the dense 2D optical flow describing the visual deformation of the input image is finally output. This model uses the 2D optical flow to deform the above-mentioned image depth abstract features, and also inputs them into the change decoding module together with the feature correlation, and decodes the binary change mask through multi-layer stacked ordinary 2D convolution. Finally, in order to solve the problem that the accuracy of steel bar inventory change detection is easily affected by the pseudo-change noise of the image background, this embodiment innovatively introduces the semantic segmentation task in the image change detection model to extract the image area related to the change to be detected in the image of the input model, that is, the semantic segmentation model is used to predict the image foreground area of interest. Specifically, this embodiment uses UNet as the structure of its segmentation decoding layer, inputs the refined image features and the image depth abstract features, and outputs a binary foreground object segmentation mask. The model applies the predicted foreground segmentation mask to the change area mask output by the change decoding layer in a pixel-by-pixel multiplication manner, so that the model pays more attention to the foreground object when detecting changes, thereby avoiding the impact of background pseudo-change noise such as pedestrians, trucks, and warehouses in the steel warehouse on image change detection. It can be understood that the above-mentioned module of this model is not specific to the steel warehouse scene. Conventional outdoor street scenes and other ordinary optical images can use this model to detect image changes, and it has good change detection accuracy and certain optical flow estimation and semantic segmentation effects.
[0106] For the image change detection model used to realize steel bar inventory change detection in this embodiment, the steel bar image change detection dataset is used for model training. In order to further improve the effect of change detection and accelerate the model fitting speed, this embodiment refers to existing research work, and adopts three large-scale outdoor street scene image datasets based on Cityscapes, ADE20K, and DPED, as well as randomly selecting foreground objects from the COCO dataset and performing image superposition, and randomly generating image transformations to construct a synthetic image pre-training dataset with change labels, semantic segmentation labels, and optical flow labels, which is used to pre-train the image change detection model of this embodiment before fine-tuning the above-mentioned steel bar image change detection dataset. Among them, the total number of iterations set for pre-training is 40, the total number of iterations set for fine-tuning is 20, the single batch size is 8, the initial learning rate of training is 0.0002, and the learning rate exponential prediction and learning rate cosine decay strategies are adopted. The loss function in this embodiment consists of three parts, and the specific formula is:
[0107] Loss = Σ i [λ Li ×(λ epe loss epe (i) +λ ce loss ce (i) +λ focal loss focal (i) )]
[0108] Among them, Loss is the total loss, loss epe (i) is the optical flow endpoint error calculated based on the optical flow estimation result and the actual label, loss ce (i) The cross entropy error calculated based on the semantic segmentation result and the actual label, loss focal (i) is the Focal error calculated based on the change detection result and the actual label, i is the number of layers in the multi-scale processing, and its value range in the model described in this embodiment is 1, 2, 3, and 4. In order to measure the weights between different loss terms, this embodiment also discloses a pair of loss weights with a certain fitting effect, and these loss weights are different from the settings used in existing research work. After experimental exploration, when λ is L1 , L2 , L3 , L4 , epe , ce , focalWhen the values are set to 0.32, 0.08, 0.12, 0.38, 0.02, 1.0, and 0.5, respectively, the image change detection model has a relatively good effect. This embodiment uses the PyTorch deep learning computing framework to train the model in the steel bar image change detection training set. Based on the image change detection model described in this embodiment, Figure 6 As shown in Figure 2, randomly selecting a steel bar pile image in the corresponding validation set can achieve good change detection results.
[0109] For the target detection model, this embodiment obtains a set of original steel bar pile images that can be used to train the target detection model from the several steel bar pile images collected above, also through image filtering, resizing, image cropping, brightness adjustment and other operations, and uses the LabelImg tool to label objects such as the end faces of the stacked steel bar bundles in the image to construct a steel bar bundle target detection dataset for model training. After data partitioning, the training set part of the dataset contains 405 images with a total of 28,274 targets, and the validation set part contains 135 images with a total of 9,541 targets, and their resolutions are all processed to 1280x720. As an example, Figure 7 Shows some images and their labels of the steel bar bundle object detection dataset.
[0110] Considering that the fitting effect and generalization effect of existing target detection models such as RCNN series and YOLO series on common target detection datasets are already high enough, this embodiment does not consider introducing improvements in model structure and loss function. This embodiment uses the PyTorch deep learning computing framework to train the YOLOv8 target detection model in the steel bar bundle target detection dataset. Based on the model, a random selection of a steel bar pile image in the corresponding verification set can obtain good target detection results.
[0111] As described in step S14 of Example 1, after the target detection model is matched with the image change detection model, the purpose of detecting changes in steel bar inventory can be achieved. There are two model matching methods here, which have been explained in Example 1 and will not be repeated here. It should be pointed out that in order to improve the efficiency of steel bar inventory change detection, facilitate the modification of inventory change detection results described in step S15 of Example 1, facilitate the implementation of related software systems, etc., more specifically, this embodiment adopts the following method to cooperate with the two models: the steel bar reference image and the steel bar target image are input into a preset image change detection model to predict the image change area where the steel bar inventory changes, and the result is usually returned in the form of a binary mask; at the same time, the steel bar target image is input into a preset target detection model to predict the steel bar bundle target in the image, and the result is usually returned in the form of target candidate box coordinates; for all the steel bar bundle candidate boxes in the steel bar target image, they are filtered according to the image change detection area binary mask, and the initial change inventory number of the inventory change in the steel bar target image can be obtained; in this way, when the operator modifies the image change detection result or the target detection result through user interaction, it only needs to re-execute the filtering operation to quickly update, and finally obtain the target change inventory number, thereby achieving the purpose of steel bar inventory change detection.
[0112] Taking into account the performance of steel bar inventory change detection, this embodiment deploys the camera shooting guidance generation model involved in the above steps to mobile terminals such as mobile phones, and taking into account the accuracy of steel bar inventory change detection, this embodiment deploys the above image change detection model and target detection model to the server end assembled with a graphics card, and implements mobile terminal system operation.
[0113] Specifically, this embodiment uses the JetBrains IntelliJ IDEA development tool to develop and implement the above system. Taking the Android mobile terminal as an example, first, in order to support the mobile terminal to perform reasoning based on the traditional computer vision algorithm to build a camera shooting guide generation model described in step S12 of Example 1, it is necessary to rely on the open source OpenCV Android SDK to implement, including using the CharucoDetector, solvePnP, detectAndCompute, findEssentialMat and other interfaces provided by it to implement the algorithm based on the feature point and the three-dimensional camera set, and obtain the relative position of the mobile terminal camera movement when shooting the steel bar pile image, so as to realize the camera shooting guide generation. Secondly, in order to support the mobile terminal to perform reasoning based on the use of artificial intelligence and computer vision algorithms to build a camera shooting guide generation model described in step S12 of Example 1, it is necessary to convert the model into a model format that can be run by the mobile terminal, such as the ONNX format. This embodiment specifically relies on the libonnxruntime.so dynamic link library to realize the reasoning of the model. In addition, this embodiment may also consider a TTS voice function module, so that after the shooting angle is generated, the interactive operation of the system can be realized by relying on the steel bar pile real-time video stream acquisition control method described in step S13 of embodiment 1, so that when acquiring the steel bar pile real-time video stream, the mobile device camera can be guided to move and shoot through voice prompts and interface displays at the same time, such as Figure 8 .
[0114] For the image change detection model and the target detection model, this embodiment deploys them on the server side equipped with a graphics card such as Linux, and specifically uses the Python programming language and the Flask development framework for development and implementation. Specifically, when the mobile terminal successfully obtains the target image of the steel pile, it will send it together with the steel pile reference image to the server program. After the server program receives the image pair, the model inference process can be started. The inference process of this embodiment is shown above and will not be repeated here. When the server successfully obtains the steel pile image change area and the steel pile inventory change detection results, Fig. 9 , the newly added, missing, and unchanged steel bar bundles will be marked and displayed in the form of target points on the Android terminal, and the inventory change detection results will be displayed in the mobile terminal program. Based on step S15 of embodiment 1, when the operator finds that the detection result has inventory omissions or false detections, such as Fig.10 ,This embodiment will allow the operator to add or delete the change detection results by clicking the error detection ,steel bundle on the screen through interface interaction, and display the change inventory results in real time in the mobile ,program as the target change inventory number.
[0115] In this embodiment, by relying on the camera shooting guidance generation model, image change detection model, and target detection model, the inventory changes of steel bars such as bars and wires can be detected more accurately, efficiently, and intelligently when there are several practical problems such as irregular steel bar stacking, complex and changeable steel bar warehouse environment, and misaligned images of manually shot steel bars, as well as several technical challenges such as the accuracy of steel bar inventory change detection is easily affected by image shooting alignment quality and image background pseudo-change noise. In this embodiment, the mobile terminal program and the server terminal program process steel bar inventory change detection system have good interactivity, and its deployment scheme and operation logic can effectively improve the efficiency and accuracy of steel bar inventory change detection in actual steel bar warehouse scenes. The steel bar inventory change detection proposed by the present invention relying on the above steps and its optional steps and system is a solution that has not been proposed or involved in existing systems and studies, and has certain innovation and practical significance.
[0116] Example 4
[0117] See also Fig.11 As shown, an embodiment of the present invention discloses a steel bar inventory change detection device, comprising:
[0118] A steel bar pile real-time video stream acquisition module 11 is used to acquire a steel bar pile real-time video stream obtained after real-time video stream acquisition of the steel bar pile;
[0119] The camera shooting guide generation module 12 is used to detect the preset steel bar pile reference image and the current latest frame of the real-time video stream of the steel bar pile using a preset camera shooting guide generation model, and obtain the camera shooting guide; the preset steel bar pile reference image is a historical state image of a specific shooting angle of the current specific steel bar pile, and the image usually has the ability to represent the inventory information of the steel bar pile;
[0120] The steel bar pile target image acquisition module 13 is used to determine whether the camera shooting guide meets the preset camera shooting angle determination condition. If the camera shooting angle is suitable, the current latest frame of the steel bar pile real-time video stream is temporarily stored and regarded as the steel bar pile target image. If the camera shooting angle is not suitable, the current shooting angle of the steel bar pile real-time video stream is adjusted based on the camera shooting guide and using the preset steel bar pile real-time video stream acquisition control method; the steel bar pile target image is an image frame in the steel bar pile real-time video stream that is close to the preset steel bar pile reference image shooting angle;
[0121] The initial change inventory number acquisition module 14 is used to detect the preset steel bar pile reference image and the steel bar pile target image using a preset image change detection model and a preset target detection model to obtain the steel bar pile image change area of the steel bar pile target image compared with the preset steel bar pile reference image, and the initial change inventory number of steel bars such as rods or wires; the steel bar pile image change area is a pixel area in the input steel bar pile image where semantic changes such as addition or deletion exist, and has the ability to characterize the steel bar inventory change between the current moment and a certain historical moment; the initial change inventory number is the actual number of steel bars detected by the target detection model where inventory changes such as addition or deletion exist in the steel bar pile image change area;
[0122] The target change inventory number acquisition module 15 is used to determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number. If they are inconsistent, the initial change inventory number is updated through a preset steel bar inventory update method to obtain a target change inventory number that is consistent with the actual change inventory number.
[0123] For the specific implementation functions, advantages and beneficial effects of module 11 and module 15, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be elaborated here.
[0124] In some specific embodiments, the steel bar pile real-time video stream acquisition module 11 may specifically include:
[0125] The video acquisition unit is used to use a camera device with image and video acquisition functions such as a mobile phone to perform real-time video acquisition of the steel bar pile to obtain an original video stream from the camera.
[0126] In some specific embodiments, the camera shooting instruction generating module 12 may specifically include:
[0127] A camera shooting guide generation model selection unit is used to select one camera shooting guide generation model with better effect in the corresponding steel bar pile shooting scene from a preset camera shooting guide generation model set;
[0128] A camera relative posture acquisition unit is used to generate the first sub-step of the model by using the camera shooting guide and inputting two steel bar pile images taken before and after to obtain the absolute posture change of the camera between the two shots, that is, the relative posture of the camera;
[0129] The camera shooting guide acquisition unit is used to use the second sub-step of the camera shooting guide generation model to post-process the camera relative posture into a camera shooting guide to prompt the operator how to adjust the camera's shooting angle of view.
[0130] In some specific embodiments, the steel bar pile target image acquisition module 13 may specifically include:
[0131] A camera shooting angle determination unit is used to determine whether the current camera shooting angle is similar to the historical angle by calculating and comparing preset thresholds such as posture physical distance according to the characteristics of the camera shooting guide generation model adopted;
[0132] The steel bar pile real-time video stream shooting angle adjustment unit is used to execute a text reminder program and / or a voice reminder program based on the camera shooting guide to perform a prompt operation for obtaining a steel bar pile image with a suitable angle of view, so as to obtain a steel bar pile image with a shooting angle closer to the prompt operation.
[0133] In some specific embodiments, the initial change inventory number acquisition module 14 may specifically include:
[0134] A steel bar pile image change region detection unit, used to train an initial model constructed based on an artificial intelligence algorithm using a pre-made image change detection training set, so as to obtain the preset image change detection model for detecting and obtaining a steel bar pile image change region;
[0135] The initial change inventory number calculation unit of steel bars is used to train the initial model constructed based on the artificial intelligence algorithm using a pre-made target detection training set to obtain the preset target detection model and calculate the initial change inventory number of steel bars such as bars or wires.
[0136] In some specific embodiments, the target change inventory number acquisition module 15 may specifically include:
[0137] A steel bar inventory consistency determination unit, configured to determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number after obtaining the initial change inventory number through the preset image change detection model and the preset target detection model;
[0138] The steel bar inventory updating unit is used to update the initial changed inventory number by using a preset steel bar inventory updating method after determining whether the preset steel bar inventory is consistent or not.
[0139] Example 5
[0140] The invention also discloses an electronic device. Fig.12 This is a structural diagram of an electronic device 20 according to this exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the scope of application of the present application.
[0141] The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the steel bar inventory change detection method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0142] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0143] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0144] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the steel bar inventory change detection method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.
[0145] Example 6
[0146] The present invention also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the above disclosed steel bar inventory change detection method. The specific steps of the method can refer to the corresponding contents disclosed in the signed embodiment, and will not be repeated here.
[0147] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0148] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0149] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0150] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0151] The above embodiments have provided a detailed introduction to the technical solutions provided by the present application, and the descriptions thereof are only used to help understand the methods and core ideas of the present application, but the implementation modes of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.
Claims
1. A method for detecting changes in steel bar inventory, It is characterized in that The following steps are involved: Get real-time video stream of steel bar pile; Detecting a preset steel bar pile reference image and a current latest frame of the steel bar pile real-time video stream to obtain a camera shooting guide; According to the camera shooting guide, it is determined whether the camera shooting angle meets the preset conditions. If so, the current latest frame of the real-time video stream of the steel bar pile is temporarily stored and regarded as the target image of the steel bar pile; if not, the current shooting angle of the real-time video stream of the steel bar pile is adjusted according to the camera shooting guide; The preset rebar pile reference image and the rebar pile target image are detected to obtain the rebar pile image change area of the rebar pile target image compared with the preset rebar pile reference image, and the initial change inventory number of the rebars; determine whether the actual change inventory number in the rebar pile image change area is consistent with the initial change inventory number; if not, update the initial change inventory number to obtain a target change inventory number that is consistent with the actual change inventory number.
2. A method for detecting steel bar inventory changes according to claim 1, It is characterized in that The method of obtaining a real-time video stream of a steel bar pile comprises: Use image acquisition equipment to collect video of the steel bar pile in real time to obtain the original video stream from the camera; By presetting the original video stream post-processing method, the original video stream from the camera is converted into a real-time video stream of the steel bar pile that meets the processing format of the subsequent steps.
3. A steel bar inventory change detection method according to claim 1, It is characterized in that The obtaining of the camera shooting guide includes: Select one of the camera shooting guide generation models with better effect in the corresponding steel bar pile shooting scene from the preset camera shooting guide generation model set; Obtain two images of the steel bar pile taken before and after, and input the selected camera shooting guidance generation model to obtain the absolute position change of the camera taken in the two shots as the relative position of the camera; The relative camera positions are post-processed into camera shooting guidelines.
4. A method for detecting steel bar inventory changes according to claim 3, It is characterized in that The camera shooting guide generation model is obtained by constructing a camera motion estimation model based on artificial intelligence and computer vision, specifically: Using the pre-collected camera motion estimation training set, the initial model built based on artificial intelligence and computer vision algorithms is trained to obtain a preset camera motion estimation model; By utilizing the preset camera motion estimation model and encapsulating the steel bar pile image detection logic and retaining a number of intermediate detection results, a camera shooting guide generation model is constructed.
5. A method for detecting steel bar inventory changes according to claim 4, It is characterized in that The method utilizes the preset camera motion estimation model and constructs a camera shooting guide generation model by encapsulating the steel bar pile image detection logic and retaining a number of intermediate detection results, including: Using a preset camera motion estimation algorithm for constructing the camera shooting guide generation model, the two images taken before and after are detected, and the relative position and posture of the camera before and after the shooting are obtained; The camera relative posture obtained by detection is analyzed and processed by using a preset camera posture post-processing method to generate a camera shooting guide that is in an inverse relationship with the camera relative posture.
6. A method for detecting steel bar inventory changes according to claim 1, It is characterized in that The step of adjusting the current shooting angle of the real-time video stream of the steel bar pile according to the camera shooting guide comprises: Based on the camera shooting guide, a text reminder program and / or a voice reminder program is executed to perform a prompt operation for obtaining the steel bar pile image with a suitable viewing angle, so as to obtain the steel bar pile image with a shooting viewing angle closer according to the prompt operation.
7. A method for detecting steel bar inventory changes according to claim 1, It is characterized in that The steel bar pile image change region is obtained by: The initial model built based on the artificial intelligence algorithm is trained using a pre-made image change detection training set to obtain an image change detection model for detecting the change area of the steel bar pile image; The initial change inventory number of the steel bars is obtained by: The initial model built based on the artificial intelligence algorithm is trained using a pre-made object detection training set to obtain an object detection model for the initial change inventory number of steel bars.
8. A method for detecting steel bar inventory changes according to claim 1, It is characterized in that The determining whether the actual change inventory number in the change area of the steel bar pile image is consistent with the initial change inventory number comprises: After obtaining the initial change inventory number, the preset steel bar inventory consistency judgment method is used to determine whether the actual change inventory number in the steel bar pile image change area is consistent with the initial change inventory number. If they are inconsistent, the preset steel bar inventory update method is used to update the initial change inventory number.
9. An electronic device, It is characterized in that include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steel bar inventory change detection method described in any one of claims 1-8.
10. A computer-readable storage medium, It is characterized in that Used to store a computer program, which, when executed by a processor, implements the steel bar inventory change detection method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Steel bar counting method, device and equipment and storage medium
CN115546221A
Method and device for detecting inventory, electronic equipment and storage medium
CN116415892A
Refrigerator inventory judgment method and device, electronic equipment and storage medium
CN117152679A
Method and system for inventory verification
US8438084B1
