Image processing method, device, equipment, storage medium and program product
By performing binarization and offset processing on the images, filtering and fitting boundary points, the problems of slow image boundary recognition speed and low accuracy in the prior art are solved, and more efficient boundary recognition and detection effects are achieved.
Patent Information
- Application Number
- CN202211165052.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-09-23
AI Technical Summary
In the image processing, the prior art, when identifying the boundaries of the target object in the image through machine learning models, the speed is slow and the accuracy is not high, and the boundary recognition is prone to problems such as inaccurate or unrecognized boundary recognition.
By performing binarization of the image to be processed and performing offset processing in at least one direction, the pixel differences between the binarized image and the offset image are compared, candidate boundary points are selected, fit and integrate, and the target object is obtained. the boundary.
It improves the recognition speed and accuracy of object boundaries in the image, avoids the increase in model training and inference time, reduces the possibility of prediction errors, and enhances the efficiency and accuracy of target object detection.
Smart Images

Figure CN117011532B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information processing technology, and in particular to an image processing method, apparatus, device, storage medium and computer program product. Background Art
[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, it enables machines to have the functions of perception, reasoning and decision-making. Artificial Intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0003] Image processing is also an important application direction of artificial intelligence. In related technologies, machine learning models are usually used to identify the boundaries of targets (i.e., objects) in images. However, 1) the inference time of the machine learning model needs to be introduced, which makes the boundary recognition time longer and slower; 2) the accuracy of extracting bounding boxes through machine learning models may introduce additional uncertainty (such as the model prediction results are prone to errors, resulting in inaccurate boundary recognition, or no boundary recognition, etc.), resulting in insufficient boundary recognition accuracy. Summary of the invention
[0004] The embodiments of the present application provide an image processing method, apparatus, device, storage medium and computer program product, which can improve the recognition efficiency and accuracy of the boundaries of objects in an image.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The present application provides an image processing method, including:
[0007] Binarization is performed on the image to be processed including the target object to obtain a binary image;
[0008] Performing an offset process on the binary image in at least one direction by an offset amount to obtain an offset image in each of the directions;
[0009] For the offset image in each of the directions, each first pixel point in the binary image is compared with a second pixel point at a corresponding position in the offset image to obtain a comparison result, and
[0010] Based on the comparison result, taking a first pixel point in the binary image that is different from a second pixel point in the offset image as a candidate boundary point of the target object;
[0011] For each of the directions, fitting the candidate boundary points in the direction to obtain a sub-boundary of the target object in the direction;
[0012] The sub-boundaries in each of the directions are integrated to obtain an object boundary of the target object in the image to be processed, and the object boundary is used for detecting the target object.
[0013] In the above scheme, when the number of the candidate boundary points is multiple, after taking the first pixel point in the binary image that is different from the second pixel point in the offset image as the candidate boundary point of the target object based on the comparison result, the method further includes:
[0014] Performing clustering processing on the plurality of candidate boundary points to obtain at least one cluster, each of the clusters including at least one candidate boundary point;
[0015] Selecting a target cluster that meets the condition from the at least one cluster, and using the candidate boundary point included in the target cluster as the first candidate boundary point;
[0016] The step of fitting the candidate boundary points in each direction to obtain the sub-boundary of the target object in the direction includes:
[0017] For each of the directions, the first candidate boundary points in the direction are fitted to obtain a sub-boundary of the target object in the direction.
[0018] In the above solution, the step of selecting a target cluster that meets the conditions from the at least one cluster includes:
[0019] Acquire the number of boundary points of the candidate boundary points included in each of the clusters, and use the clusters whose number of boundary points meets the quantity condition as the target clusters; or,
[0020] An inter-cluster distance between any two clusters in the at least one cluster is obtained, and a cluster whose inter-cluster distance satisfies a distance condition is used as the target cluster.
[0021] In the above solution, there are multiple candidate boundary points in the direction, and fitting the candidate boundary points in the direction to obtain the sub-boundary of the target object in the direction includes:
[0022] Sampling a plurality of second candidate boundary points from a plurality of candidate boundary points in the direction, and performing straight line fitting on the plurality of second candidate boundary points to obtain corresponding straight lines;
[0023] Determine the distance between each of the candidate boundary points and the straight line, and determine the number of boundary points of the candidate boundary points whose distance is lower than a distance threshold;
[0024] The above operation is executed repeatedly until the iteration process ends;
[0025] Determine the maximum number of boundary points among the numbers of boundary points corresponding to the multiple iteration processes, and determine the target straight line obtained by the target iteration process corresponding to the maximum number of boundary points;
[0026] The target straight line is determined as a sub-boundary of the target object in the direction.
[0027] In the above scheme, when the sub-boundaries in the directions include a left sub-boundary, a right sub-boundary, an upper sub-boundary and a lower sub-boundary, integrating the sub-boundaries in each of the directions to obtain the object boundary of the target object in the image to be processed includes:
[0028] Determine the first coordinate of the intersection point between the straight line corresponding to the left sub-boundary and the straight line corresponding to the upper sub-boundary, and
[0029] Determine a second coordinate of an intersection point between the straight line corresponding to the right sub-boundary and the straight line corresponding to the lower sub-boundary;
[0030] Based on the straight lines corresponding to the sub-boundaries in each of the directions, in combination with the first coordinate and the second coordinate, an object boundary of the target object in the image to be processed is obtained.
[0031] In the above solution, after integrating the sub-boundaries in each of the directions to obtain the object boundary of the target object in the image to be processed, the method further includes:
[0032] Determining whether the object boundary satisfies a boundary anomaly condition of the target object;
[0033] When the object boundary satisfies the boundary abnormality condition, abnormality prompt information is output, and the abnormality prompt information is used to prompt that the target object has an abnormality.
[0034] The present application also provides an image processing device, including:
[0035] A binarization module, used for performing binarization processing on the image to be processed including the target object to obtain a binarized image;
[0036] An offset module, used for performing an offset process on the binary image in at least one direction by an offset amount to obtain an offset image in each of the directions;
[0037] a comparison module, for comparing, for the offset images in each of the directions, each first pixel point in the binary image with a second pixel point at a corresponding position in the offset image to obtain a comparison result, and based on the comparison result, taking a first pixel point in the binary image that is different from a second pixel point in the offset image as a candidate boundary point of the target object;
[0038] A fitting module, used for fitting the candidate boundary points in each direction to obtain a sub-boundary of the target object in the direction;
[0039] The integration module is used to integrate the sub-boundaries in each of the directions to obtain the object boundary of the target object in the image to be processed, and the object boundary is used to detect the target object.
[0040] In the above scheme, the binarization module is also used to obtain the pixel value of each pixel in the image to be processed; the pixel whose pixel value exceeds the pixel threshold is used as the third pixel, and the pixel whose pixel value does not exceed the pixel threshold is used as the fourth pixel; the pixel grayscale value of the third pixel is determined as the first grayscale value, and the pixel grayscale value of the fourth pixel is determined as the second grayscale value, so as to obtain the binary image.
[0041] In the above scheme, the binarization module is also used to perform the following processing on each pixel point in the image to be processed: obtain the channel pixel value of the pixel point on the red R, green G, and blue B color channels respectively; average the channel pixel values on multiple color channels to obtain an average processing result, and use the average processing result as the pixel value of the pixel point.
[0042] In the above scheme, the offset module is also used to offset the binary image by one pixel in the horizontal direction according to a first offset direction to obtain the offset image in the horizontal direction; and to offset the binary image by one pixel in the vertical direction according to a second offset direction to obtain the offset image in the vertical direction.
[0043] In the above scheme, the comparison module is also used to perform the following processing for each of the first pixel points in the binary image: determine the pixel value difference between the pixel value of the first pixel point and the pixel value of the second pixel point at the corresponding position in the offset image; when the pixel value difference is not 0, obtain a comparison result that there is a difference between the first pixel point and the second pixel point at the corresponding position in the offset image.
[0044] In the above scheme, the comparison module is also used to perform an offset processing on the binary image in the horizontal direction according to the offset direction from left to right, so as to obtain the offset image in the horizontal direction; the comparison module is also used to use the first pixel point whose pixel value difference is greater than 0 as the left candidate boundary point of the target object, and use the first pixel point whose pixel value difference is less than 0 as the right candidate boundary point of the target object; the fitting module is also used to fit the left candidate boundary point in the horizontal direction to obtain the left sub-boundary of the target object in the horizontal direction, and to fit the right candidate boundary point in the horizontal direction to obtain the right sub-boundary of the target object in the horizontal direction.
[0045] In the above scheme, the comparison module is also used to offset the binary image in the vertical direction by an offset amount according to the offset direction from top to bottom, so as to obtain the offset image in the vertical direction; the comparison module is also used to use the first pixel point whose pixel value difference is greater than 0 as the upper candidate boundary point of the target object, and use the first pixel point whose pixel value difference is less than 0 as the lower candidate boundary point of the target object; the fitting module is also used to fit the upper candidate boundary point in the vertical direction to obtain the upper sub-boundary of the target object in the vertical direction, and to fit the lower candidate boundary point in the vertical direction to obtain the lower sub-boundary of the target object in the vertical direction.
[0046] In the above scheme, when the number of the candidate boundary points is multiple, the comparison module is also used to cluster the multiple candidate boundary points to obtain at least one cluster cluster, each of which includes at least one candidate boundary point; a target cluster cluster that meets the conditions is selected from the at least one cluster cluster, and the candidate boundary points included in the target cluster cluster are used as the first candidate boundary points; the fitting module is also used to fit the first candidate boundary points in the direction for each of the directions to obtain the sub-boundary of the target object in the direction.
[0047] In the above scheme, the comparison module is also used to obtain the number of boundary points of the candidate boundary points included in each of the clustering clusters, and use the clustering cluster whose number of boundary points meets the quantity condition as the target clustering cluster; or, obtain the inter-cluster distance between any two clustering clusters in at least one clustering cluster, and use the clustering cluster whose inter-cluster distance meets the distance condition as the target clustering cluster.
[0048] In the above scheme, when the number of the candidate boundary points is multiple and the direction is a horizontal direction, the comparison module is also used to perform the following processing starting from the first pixel point of the first row of the binary image: for the current row to be scanned, according to the first scanning direction, starting from the first first pixel point of the current row, the first pixel point of the current row is scanned in sequence; when one of the multiple candidate boundary points is scanned, the scanned candidate boundary point is used as the candidate boundary point in the horizontal direction; determine whether there is a target row that is a preset row step value away from the current row; if the target row exists, jump from the current row to the target row to use the target row as the current row to be scanned, and repeat the above operations; if the target row does not exist, execute the end operation.
[0049] In the above scheme, when the number of the candidate boundary points is multiple and the direction is the vertical direction, the comparison module is also used to perform the following processing starting from the first pixel point of the first column of the binary image: for the current column to be scanned, according to the second scanning direction, starting from the first first pixel point of the current column, the first pixel points of the current column are scanned in sequence; when one candidate boundary point among the multiple candidate boundary points is scanned, the scanned candidate boundary point is used as the candidate boundary point in the vertical direction; determine whether there is a target column that is a preset column step value away from the current column; if the target column exists, jump from the current column to the target column to use the target column as the current column to be scanned, and repeat the above operations; if the target column does not exist, perform the end operation.
[0050] In the above scheme, there are multiple candidate boundary points in the direction, and the fitting module is also used to sample multiple second candidate boundary points from the multiple candidate boundary points in the direction, and perform straight line fitting on the multiple second candidate boundary points to obtain corresponding straight lines; determine the distance between each of the candidate boundary points and the straight line, and determine the number of boundary points of the candidate boundary points whose distance is lower than the distance threshold; loop through the above operations until the iterative process ends; determine the maximum number of boundary points corresponding to multiple iterative processes, and determine the target straight line obtained by the target iterative process corresponding to the maximum number of boundary points; determine the target straight line as the sub-boundary of the target object in the direction.
[0051] In the above scheme, when the sub-boundaries in the direction include a left sub-boundary, a right sub-boundary, an upper sub-boundary and a lower sub-boundary, the integration module is also used to determine the first coordinates of the intersection of the straight line corresponding to the left sub-boundary and the straight line corresponding to the upper sub-boundary, and to determine the second coordinates of the intersection of the straight line corresponding to the right sub-boundary and the straight line corresponding to the lower sub-boundary; based on the straight lines corresponding to the sub-boundaries in each of the directions, combined with the first coordinates and the second coordinates, the object boundary of the target object in the image to be processed is obtained.
[0052] In the above scheme, the integration module is also used to determine whether the object boundary meets the boundary abnormality condition of the target object; when the object boundary meets the boundary abnormality condition, output abnormal prompt information, and the abnormal prompt information is used to prompt that the target object has an abnormality.
[0053] The present application also provides an electronic device, including:
[0054] A memory for storing computer executable instructions;
[0055] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.
[0056] The embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the image processing method provided by the embodiment of the present application is implemented.
[0057] The embodiment of the present application also provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the image processing method provided in the embodiment of the present application.
[0058] The embodiments of the present application have the following beneficial effects:
[0059] First, a binarization process is performed on the image to be processed including the target object to obtain a binarized image; then, the binarized image is offset in at least one direction by an offset amount to obtain offset images in each direction; then, for the offset images in each direction, each first pixel point in the binarized image is compared with the second pixel point at the corresponding position in the offset image to obtain a comparison result, and based on the comparison result, the first pixel point in the binarized image that is different from the second pixel point in the offset image is used as a candidate boundary point of the target object; thus, for each direction, the candidate boundary points in the direction are fitted to obtain the sub-boundary of the target object in the direction; finally, the sub-boundaries in each direction are integrated to obtain the object boundary of the target object in the image to be processed. In this way, the boundary of the object in the image can be recognized without using a machine learning model, avoiding the introduction of model training time and inference time, and can improve the recognition speed and efficiency of the boundary of the object in the image; at the same time, it also avoids the problem of errors in the model prediction results, and can improve the recognition accuracy of the boundary of the object in the image; and the object boundary can be used for detecting the target object, which can further improve the detection efficiency and detection accuracy of the target object detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 is a schematic diagram of the architecture of an image processing system 100 provided in an embodiment of the present application;
[0061] Figure 2 is a structural schematic diagram of an electronic device 500 for implementing an image processing method provided in an embodiment of the present application;
[0062] Figure 3 is a flowchart of an image processing method provided in an embodiment of the present application;
[0063] Figure 4 is a schematic diagram of a binarized image provided in an embodiment of the present application;
[0064] Figure 5 is a schematic diagram of the first candidate boundary point on the left side provided in an embodiment of the present application;
[0065] Figure 6 is a schematic diagram of left candidate boundary points in the horizontal direction provided by an embodiment of the present application;
[0066] Figure 7 is a schematic diagram of candidate boundary points in the horizontal direction provided by an embodiment of the present application;
[0067] Figure 8 is a schematic diagram of candidate boundary points in the vertical direction provided by an embodiment of the present application;
[0068] Fig. 9is a schematic diagram of a straight line corresponding to the left sub-boundary provided in an embodiment of the present application;
[0069] Fig.10 is a schematic diagram of an object boundary of a target object provided in an embodiment of the present application;
[0070] Fig.11 It is a flowchart of the image processing method provided in the embodiment of the present application. DETAILED DESCRIPTION
[0071] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0072] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0073] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0075] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0076] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.
[0077] 2) Region of interest (ROI): In machine vision and image processing, ROI is the area to be processed that is outlined in the image to be processed in the form of a box, circle, ellipse, irregular polygon, etc. ROI is the focus of attention in the image processing process. By defining ROI, the image processing time can be reduced and the accuracy of image processing can be increased.
[0078] 3) Bounding box: This is the object boundary mentioned above. In object detection, a bounding box is usually used to describe the location of the object. In some examples, the bounding box can be a rectangular box, which can be determined by the coordinates of the upper left corner vertex (x1, y1) and the coordinates of the lower right corner vertex (x2, y2) of the rectangular box.
[0079] The embodiments of the present application provide an image processing method, apparatus, device, storage medium and computer program product, which can improve the recognition efficiency and accuracy of the boundaries of objects in an image.
[0080] The following describes the implementation scenario of the image processing method provided by the embodiment of the present application. Figure 1 , Figure 1 It is a schematic diagram of the architecture of the image processing system 100 provided in an embodiment of the present application. To support an exemplary application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and wireless or wired links are used to realize data transmission.
[0081] The terminal 400 is used to send an image processing request to the server 200 in response to an image processing instruction for an image to be processed; the server 200 is used to receive the image processing request sent by the terminal 400; in response to the image processing request, the image to be processed including the target object is binarized to obtain a binarized image; the binarized image is offset by an offset in at least one direction to obtain an offset image in each of the directions; for the offset image in each of the directions, each first pixel in the binarized image is compared with a second pixel at a corresponding position in the offset image to obtain a comparison image. The comparison result is obtained, and based on the comparison result, a first pixel point in the binary image that is different from a second pixel point in the offset image is used as a candidate boundary point of the target object; for each of the directions, the candidate boundary points in the direction are fitted to obtain a sub-boundary of the target object in the direction; the sub-boundaries in each of the directions are integrated to obtain an object boundary of the target object in the image to be processed; based on the object boundary, the target object is detected to obtain a detection result, and the detection result is returned to the terminal 400; the terminal 400 is used to receive the detection result and display it for viewing by a user.
[0082] In some embodiments, the image processing method provided in the embodiments of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, it can also be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. For example, the terminal alone executes the image processing method provided in the embodiments of the present application, or the terminal sends an image processing request to the server, and the server executes the image processing method provided in the embodiments of the present application according to the received image processing request. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.
[0083] In some embodiments, the electronic device for implementing image processing provided in the embodiments of the present application may be various types of terminal devices or servers. Among them, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal (e.g., terminal 400) may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device (e.g., an intelligent speaker), a smart home appliance (e.g., a smart TV), a smart watch, a car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application do not limit this.
[0084] In some embodiments, the image processing method provided in the embodiments of the present application can be implemented with the aid of cloud technology (Cloud Technology), which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks in a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model application, which can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources. As an example, a server (such as server 200) can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0085] In some embodiments, the terminal or server can implement the image processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.
[0086] In some embodiments, multiple servers may form a blockchain, and the servers are nodes on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the image processing method provided in the embodiment of the present application (such as the image to be processed, the binary image, the offset image, the candidate boundary points of the target object, the object boundary of the target object, the sub-boundaries of the target object in each direction, etc.) may be stored on the blockchain.
[0087] The electronic device for implementing the image processing method provided by the embodiment of the present application is described below. Figure 2 , Figure 2 Schematic diagram of the structure of an electronic device 500 for implementing an image processing method provided in an embodiment of the present application. Figure 1 Taking the server shown in the figure as an example, the electronic device 500 for implementing the image processing method provided in the embodiment of the present application includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .
[0088] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0089] The memory 550 may be removable, non-removable, or a combination thereof. The memory 550 may optionally include one or more storage devices physically located away from the processor 510. The memory 550 may include a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.
[0090] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0091] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0092] A network communication module 552, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB), etc.;
[0093] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 An image processing device 553 stored in a memory 550 is shown, which may be software in the form of a program and a plug-in, etc., and includes the following software modules: a binarization module 5531, an offset module 5532, a comparison module 5533, a fitting module 5534 and an integration module 5535. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.
[0094] The following describes the image processing method provided by the embodiment of the present application. In some embodiments, the image processing method provided by the embodiment of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, it can also be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. Taking the server implementation as an example, see Figure 3 , Figure 3 : is a flow chart of an image processing method provided in an embodiment of the present application. The image processing method provided in an embodiment of the present application includes:
[0095] Step 101: The server performs binarization processing on the image to be processed including the target object to obtain a binarized image.
[0096] In practical applications, when using machine learning models to detect objects in images (such as detecting whether the appearance of a product (i.e., an object) in an image is qualified or defective), the size of the image to be processed is relatively sensitive. Large-size images will not only slow down the detection speed, but also lead to inaccurate target detection. Therefore, the effective area of the object (i.e., ROI) can be identified in advance by performing ROI recognition on the image to be processed, so that in the subsequent object detection process, only the effective area of the object can be detected, thereby improving the object detection speed and object detection accuracy.
[0097] When the effective region (i.e., ROI) of an object is identified, the object boundary of the object in the image to be processed is actually identified, and the area surrounded by the object boundary is the effective region of the object. In an embodiment of the present application, a method for identifying the object boundary of an object in an image to be processed is provided, which will be described in detail below. In an embodiment of the present application, when the server performs object boundary identification, it first performs binarization processing on the image to be processed including the target object to obtain a binarized image.
[0098] In some embodiments, the server may perform binarization processing on the image to be processed including the target object to obtain a binary image in the following manner: obtain the pixel value of each pixel in the image to be processed; take the pixel whose pixel value exceeds the pixel threshold as the third pixel, and take the pixel whose pixel value does not exceed the pixel threshold as the fourth pixel; determine the pixel grayscale value of the third pixel as the first grayscale value, and determine the pixel grayscale value of the fourth pixel as the second grayscale value to obtain a binary image.
[0099] In some embodiments, the server can obtain the pixel value of each pixel in the image to be processed in the following manner: for each pixel in the image to be processed, the following processing is performed respectively: obtain the channel pixel value of the pixel on the red R, green G, and blue B color channels respectively; average the channel pixel values on multiple color channels to obtain an average processing result, and use the average processing result as the pixel value of the pixel.
[0100] In practical applications, first obtain the channel pixel values of the pixel points on the red R, green G, and blue B color channels respectively, then average the channel pixel values on multiple color channels to obtain the average processing result, and use the average processing result as the pixel value of the pixel point. When performing binarization processing, a pixel threshold can be set in advance, so that after obtaining the pixel value of each pixel point in the image to be processed, the pixel value of each pixel point is compared with the pixel threshold, so that the pixel point whose pixel value exceeds the pixel threshold is taken as the third pixel point, and the pixel grayscale value of the third pixel point is determined as the first grayscale value (such as 255), and the pixel point whose pixel value does not exceed the pixel threshold is taken as the fourth pixel point, and the pixel grayscale value of the fourth pixel point is determined as the second grayscale value (such as 0). In this way, a binary image of the image to be processed is obtained. As an example, see Figure 4 , Figure 4 is a schematic diagram of a binary image provided by an embodiment of the present application. Here, the binary image includes pixels with a grayscale value of 255 and pixels with a grayscale value of 0.
[0101] Step 102: performing an offset process on the binary image in at least one direction by an offset amount to obtain offset images in each direction.
[0102] In step 102, the offset may be pre-set, and the offset may be one pixel; the direction may include a horizontal direction and a vertical direction; during the offset processing, the offset processing may be performed according to a preset offset direction, for example, in the horizontal direction, the offset direction may be from left to right or from right to left; in the vertical direction, the offset direction may be from top to bottom or from bottom to top. That is, the server may perform an offset processing of an offset in at least one direction of the binary image in the following manner to obtain offset images in each direction: according to a first offset direction (such as from left to right), the binary image is offset by one pixel in the horizontal direction to obtain an offset image in the horizontal direction; and, according to a second offset direction (such as from top to bottom), the binary image is offset by one pixel in the vertical direction to obtain an offset image in the vertical direction.
[0103] Step 103: For the offset image in each direction, each first pixel point in the binary image is compared with the second pixel point at the corresponding position in the offset image to obtain a comparison result, and based on the comparison result, the first pixel point in the binary image that is different from the second pixel point in the offset image is used as a candidate boundary point of the target object.
[0104] In step 103, after obtaining the offset images in each direction, the following processing is performed for the offset images in each direction: each first pixel in the binary image is compared with the second pixel at the corresponding position in the offset image to obtain a comparison result, and based on the comparison result, the first pixel in the binary image that is different from the second pixel in the offset image is used as a candidate boundary point of the target object. In some embodiments, the server can compare each first pixel in the binary image with the second pixel at the corresponding position in the offset image in the following manner to obtain a comparison result: for each first pixel in the binary image, the following processing is performed: determine the pixel value difference between the pixel value of the first pixel and the pixel value of the second pixel at the corresponding position in the offset image; when the pixel value difference is not 0, obtain a comparison result that there is a difference between the first pixel and the second pixel at the corresponding position in the offset image; when the pixel value difference is 0, obtain a comparison result that there is no difference between the first pixel and the second pixel at the corresponding position in the offset image. Therefore, based on the comparison result, a first pixel point in the binary image that is different from a second pixel point in the offset image is used as a candidate boundary point of the target object.
[0105] In practical applications, after determining the candidate boundary points of the target object, if there are multiple candidate boundary points, some candidate boundary points can be further screened from the multiple candidate boundary points to reduce the amount of calculation for subsequent processing, improve the calculation speed, and reduce the waste of calculation resources. In this way, when there are multiple candidate boundary points, based on the comparison result, after taking the first pixel point in the binary image that is different from the second pixel point in the offset image as the candidate boundary point of the target object, the server can screen the candidate boundary points again in the following manner: clustering the multiple candidate boundary points to obtain at least one cluster, each of which includes at least one candidate boundary point; selecting a target cluster that meets the conditions from the at least one cluster, and taking the candidate boundary point included in the target cluster as the first candidate boundary point; correspondingly, for each direction, the server can fit the candidate boundary points in the direction in the following manner to obtain the sub-boundary of the target object in the direction: for each direction, fitting the first candidate boundary point in the direction to obtain the sub-boundary of the target object in the direction.
[0106] In practical applications, the multiple candidate boundary points include left candidate boundary points, right candidate boundary points, upper candidate boundary points, and lower candidate boundary points. When the number of candidate boundary points on each side is multiple, the above processing can be performed on each candidate boundary point, that is, the following processing is performed on the multiple candidate boundary points on each side: cluster the multiple candidate boundary points on the side to obtain at least one cluster, each cluster including at least one candidate boundary point; select a target cluster that meets the conditions from at least one cluster, and use the candidate boundary points included in the target cluster as the first candidate boundary point on the side; accordingly, for each direction, the server can fit the candidate boundary points in the direction in the following way to obtain the sub-boundary of the target object in the direction: for each direction, fit the first candidate boundary point on the side in the direction to obtain the sub-boundary of the target object on the side in the direction, such as the upper boundary, the lower boundary, the left boundary, and the right boundary.
[0107] As an example, see Figure 5 , Figure 5 is a schematic diagram of the first candidate boundary point on the left provided in an embodiment of the present application. Here, Figure 5 As shown in (1), the clusters obtained by clustering the candidate boundary points on the left include cluster 1, cluster 2, cluster 3 and cluster 4. Figure 5 As shown in (2), target clusters that meet the conditions are selected from multiple clusters, namely cluster 1 and cluster 3.
[0108] In some embodiments, the server may select a target cluster that meets the conditions from at least one cluster in the following manner: obtain the number of boundary points of the candidate boundary points included in each cluster, and use the cluster whose number of boundary points meets the quantity condition as the target cluster; or obtain the inter-cluster distance between any two clusters in at least one cluster, and use the cluster whose inter-cluster distance meets the distance condition as the target cluster. Exemplarily, the clustering process may be implemented using the K-means clustering algorithm: the K-means clustering algorithm may be used to specify in advance the number m of clusters obtained by clustering, and the number m may be determined by the shape of the target object, for example, m=2n+1 (n is the number of protruding parts of the target object), that is, if there is only one protruding part, m may be set to 3, and if there are two protruding parts, m may be set to 5. When selecting a target cluster that meets the conditions from multiple clusters, 1) (n+1) clusters with closer inter-cluster distances (such as (n+1) clusters with inter-cluster distances lower than the distance threshold, and (n+1) clusters with inter-cluster distances ranked at the top in ascending order) can be selected from multiple clusters as target clusters. For example, if there is only one protruding part (n=1), m can be set to 3, and after clustering, two clusters with similar inter-cluster distances can be selected as target clusters; if there are two protruding parts (n=2), m can be set to 5, and after clustering, three clusters with similar inter-cluster distances can be selected as target clusters. 2) It is also possible to select from multiple clusters the number of boundary points of candidate boundary points that meets the quantity condition (such as several clusters with the number of boundary points greater than the quantity threshold, or several clusters with the number of boundary points ranked at the top in ascending order) as target clusters.
[0109] In practical applications, after determining the candidate boundary points of the target object, if there are multiple candidate boundary points, some candidate boundary points can be further screened from the multiple candidate boundary points to reduce the amount of calculation for subsequent processing, improve the calculation speed, and reduce the waste of computing resources. Thus, in some embodiments, when there are multiple candidate boundary points and the direction is horizontal, after the first pixel point in the binary image that is different from the second pixel point in the offset image is used as a candidate boundary point of the target object based on the comparison result, the server can screen the candidate boundary points again in the following manner: starting from the first pixel point of the first row of the binary image, perform the following processing: for the current row to be scanned, according to the first scanning direction, starting from the first first pixel point of the current row, the first pixel point of the current row is scanned in sequence; when one of the multiple candidate boundary points is scanned, the scanned candidate boundary point is used as a candidate boundary point in the horizontal direction; determine whether there is a target row that is a preset row step value away from the current row; if the target row exists, jump from the current row to the target row to use the target row as the current row to be scanned, and execute the above operations in a loop; if the target row does not exist, execute the end operation.
[0110] In practical applications, the multiple candidate boundary points include left candidate boundary points and right candidate boundary points in the horizontal direction. For the binary image with the determined candidate boundary point set, starting from the first pixel point in the first row, scan each first pixel point from left to right in the current row. When a candidate boundary point is scanned, the scanned candidate boundary point is used as the left candidate boundary point in the horizontal direction; and determine whether there is a target row that is a preset row step value (such as 10 rows) away from the current row; if the target row exists, jump from the current row to the target row to use the target row as the current row to be scanned, and execute the above operation in a loop; if the target row does not exist, execute the end operation. In this way, the left candidate boundary point of the target object in the horizontal direction is obtained. As an example, see Figure 6 , Figure 6 It is a schematic diagram of left candidate boundary points in the horizontal direction provided by an embodiment of the present application.
[0111] For the binary image with the determined candidate boundary point set, starting from the first pixel point in the first row, scan each first pixel point from left to right in the current row. When a candidate boundary point is scanned, the scanned candidate boundary point is used as the right candidate boundary point in the horizontal direction; and determine whether there is a target row with a preset row step value (such as 10 rows) away from the current row; if the target row exists, jump from the current row to the target row, so as to use the target row as the current row to be scanned, and repeat the above operation; if the target row does not exist, perform the end operation. In this way, the right candidate boundary point of the target object in the horizontal direction is obtained.
[0112] In practical applications, after determining the candidate boundary points of the target object, if there are multiple candidate boundary points, some candidate boundary points can be further screened from the multiple candidate boundary points to reduce the amount of calculation for subsequent processing, improve the calculation speed, and reduce the waste of calculation resources. In this way, in some embodiments, when there are multiple candidate boundary points and the direction is vertical, after the first pixel point in the binary image that is different from the second pixel point in the offset image is used as the candidate boundary point of the target object based on the comparison result, the server can screen the candidate boundary points again in the following manner: for the current column to be scanned, according to the second scanning direction, starting from the first first pixel point of the current column, the first pixel point of the current column is scanned in sequence; when one of the multiple candidate boundary points is scanned, the scanned candidate boundary point is used as the candidate boundary point in the vertical direction; determine whether there is a target column that is a preset column step value away from the current column; if the target column exists, jump from the current column to the target column to use the target column as the current column to be scanned, and repeat the above operation; if the target column does not exist, perform the end operation.
[0113] In practical applications, the multiple candidate boundary points include upper candidate boundary points and lower candidate boundary points in the horizontal direction. For the binary image with the determined candidate boundary point set, starting from the first pixel point in the first column, in the current column, each first pixel point is scanned from top to bottom. When a candidate boundary point is scanned, the scanned candidate boundary point is used as the upper candidate boundary point in the vertical direction; and it is determined whether there is a target column with a preset column step value (such as 10 columns) away from the current column; if the target column exists, jump from the current column to the target column to use the target column as the current column to be scanned, and repeat the above operation; if the target column does not exist, the end operation is performed. In this way, the upper candidate boundary point of the target object in the vertical direction is obtained.
[0114] For the binary image with the determined candidate boundary point set, starting from the first pixel point in the first column, in the current column, scan each first pixel point from bottom to top, and when a candidate boundary point is scanned, the scanned candidate boundary point is used as the lower candidate boundary point in the vertical direction; and determine whether a target column with a preset column step value (such as 10 columns) away from the current column exists; if the target column exists, jump from the current column to the target column, so as to use the target column as the current column to be scanned, and repeat the above operation; if the target column does not exist, execute the end operation. In this way, the lower candidate boundary point of the target object in the vertical direction is obtained.
[0115] Step 104: For each direction, fit the candidate boundary points in the direction to obtain the sub-boundary of the target object in the direction.
[0116] In some embodiments, the server may perform an offset processing on the binary image in at least one direction by an offset amount to obtain offset images in each direction in the following manner: according to the offset direction from left to right, the binary image is offset by an offset amount in the horizontal direction to obtain an offset image in the horizontal direction; accordingly, the server may use the first pixel point in the binary image that is different from the second pixel point in the offset image as a candidate boundary point of the target object based on the comparison result in the following manner: use the first pixel point with a pixel value difference (such as a gray value difference) greater than 0 as the left candidate boundary point of the target object, and use the first pixel point with a pixel value difference less than 0 as the right candidate boundary point of the target object; accordingly, the server may fit the candidate boundary points in the direction in the following manner to obtain the sub-boundary of the target object in the direction: fit the left candidate boundary points in the horizontal direction to obtain the left sub-boundary of the target object in the horizontal direction, and fit the right candidate boundary points in the horizontal direction to obtain the right sub-boundary of the target object in the horizontal direction.
[0117] In practical applications, the binary image is offset by one pixel in the horizontal direction according to the offset direction from left to right to obtain an offset image in the horizontal direction; for each first pixel in the binary image, the pixel grayscale value of the first pixel is subtracted from the pixel grayscale value of the second pixel at the corresponding position in the offset image to obtain a grayscale value difference; the target first pixel corresponding to the grayscale value difference that is not 0 is determined as a plurality of candidate boundary points of the target object in the horizontal direction. Accordingly, the first pixel with a positive grayscale value difference (such as 255) is determined to be a candidate boundary point on the left, and the first pixel with a negative grayscale value difference (such as -255) is determined to be a candidate boundary point on the right. As an example, see Figure 7 , Figure 7 is a schematic diagram of candidate boundary points in the horizontal direction provided by an embodiment of the present application. Here, the candidate boundary points in the horizontal direction include left candidate boundary points and right candidate boundary points.
[0118] Correspondingly, when fitting the candidate boundary points in the direction, the left candidate boundary points in the horizontal direction are fitted to obtain the left sub-boundary of the target object in the horizontal direction, and the right candidate boundary points in the horizontal direction are fitted to obtain the right sub-boundary of the target object in the horizontal direction.
[0119] In some embodiments, the server may perform an offset processing on the binary image in at least one direction by an offset amount to obtain offset images in each direction in the following manner: according to the offset direction from top to bottom, the binary image is offset by an offset amount in the vertical direction to obtain an offset image in the vertical direction; accordingly, the server may use the first pixel point in the binary image that is different from the second pixel point in the offset image as a candidate boundary point of the target object based on the comparison result in the following manner: use the first pixel point whose pixel value difference (such as gray value difference) is greater than 0 as the upper candidate boundary point of the target object, and use the first pixel point whose pixel value difference is less than 0 as the lower candidate boundary point of the target object; accordingly, the server may fit the candidate boundary points in the direction in the following manner to obtain the sub-boundary of the target object in the direction: fit the upper candidate boundary points in the vertical direction to obtain the upper sub-boundary of the target object in the vertical direction, and fit the lower candidate boundary points in the vertical direction to obtain the lower sub-boundary of the target object in the vertical direction.
[0120] In practical applications, the binary image is offset by one pixel in the vertical direction according to the offset direction from top to bottom to obtain an offset image in the vertical direction; for each first pixel in the binary image, the pixel grayscale value of the first pixel is subtracted from the pixel grayscale value of the second pixel at the corresponding position in the offset image to obtain a grayscale value difference; the target first pixel (i.e., the difference pixel) corresponding to the grayscale value difference that is not 0 is determined as a plurality of candidate boundary points of the target object in the vertical direction. Among them, the first pixel with a positive grayscale value difference (such as 255) is determined to be the upper candidate boundary point, and the first pixel with a negative grayscale value difference (such as -255) is determined to be the lower candidate boundary point. As an example, see Figure 8 , Figure 8 is a schematic diagram of candidate boundary points in the vertical direction provided by an embodiment of the present application. Here, the candidate boundary points in the vertical direction include upper candidate boundary points and lower candidate boundary points.
[0121] Correspondingly, when fitting the candidate boundary points in the direction, the upper candidate boundary points in the vertical direction are fitted to obtain the upper sub-boundary of the target object in the vertical direction, and the lower candidate boundary points in the vertical direction are fitted to obtain the lower sub-boundary of the target object in the vertical direction.
[0122] In some embodiments, there are multiple candidate boundary points in a direction, and the server can fit the candidate boundary points in the direction in the following manner to obtain a sub-boundary of the target object in the direction: sample multiple second candidate boundary points from the multiple candidate boundary points in the direction, and perform straight line fitting on the multiple second candidate boundary points to obtain corresponding straight lines; determine the distance between each candidate boundary point and the straight line, and determine the number of boundary points of the candidate boundary points whose distance is lower than a distance threshold; perform the above operations in a loop until the iteration ends; determine the maximum number of boundary points corresponding to multiple iteration processes, and determine the target straight line obtained by the target iteration process corresponding to the maximum number of boundary points; determine the target straight line as the sub-boundary of the target object in the direction.
[0123] In practical applications, the direction includes a horizontal direction and a vertical direction, the horizontal direction includes a left sub-boundary and a right sub-boundary, and the vertical direction includes an upper sub-boundary and a lower sub-boundary. For the horizontal direction, the left candidate boundary points in the horizontal direction may be fitted to obtain the left sub-boundary of the target object in the horizontal direction, and the right candidate boundary points in the horizontal direction may be fitted to obtain the right sub-boundary of the target object in the horizontal direction. For the vertical direction, the upper candidate boundary points in the vertical direction may be fitted to obtain the upper sub-boundary of the target object in the vertical direction, and the lower candidate boundary points in the vertical direction may be fitted to obtain the lower sub-boundary of the target object in the vertical direction.
[0124] The candidate boundary points on each side can be fitted in the following way to obtain the corresponding sub-boundary: from the multiple candidate boundary points on this side, sample multiple second candidate boundary points, and perform straight line fitting on the multiple second candidate boundary points to obtain the corresponding straight line; determine the distance between each candidate boundary point and the straight line, and determine the number of boundary points of the candidate boundary points whose distance is lower than the distance threshold; repeat the above operation until the iteration ends; determine the maximum number of boundary points corresponding to multiple iterations, and determine the target straight line obtained by the target iteration process corresponding to the maximum number of boundary points; determine the target straight line as the sub-boundary of the target object on this side. As an example, see Fig. 9 , Fig. 9 It is a schematic diagram of the straight line corresponding to the left sub-boundary provided in an embodiment of the present application.
[0125] Step 105: Integrate the sub-boundaries in each direction to obtain the object boundary of the target object in the image to be processed.
[0126] The object boundary is used to detect the target object.
[0127] In practical applications, when the sub-boundaries in a direction include a left sub-boundary, a right sub-boundary, an upper sub-boundary, and a lower sub-boundary, the server can integrate the sub-boundaries in each direction in the following manner to obtain the object boundary of the target object in the image to be processed: determine the first coordinate of the intersection point between the straight line corresponding to the left sub-boundary and the straight line corresponding to the upper sub-boundary, and determine the second coordinate of the intersection point between the straight line corresponding to the right sub-boundary and the straight line corresponding to the lower sub-boundary; based on the straight lines corresponding to the sub-boundaries in each direction, combined with the first coordinate and the second coordinate, obtain the object boundary of the target object in the image to be processed.
[0128] After calculating the mathematical models (y=kx+b) corresponding to the multiple sub-boundaries of the target object (including the upper boundary, lower boundary, left boundary and right boundary), the first coordinate of the upper left intersection point and the second coordinate of the lower right intersection point are calculated and output through the intersection of the two lines, and the slopes and intercepts of the straight lines corresponding to the four sub-boundaries are output, that is, k and b in y=kx+b. Based on the output first coordinate of the upper left intersection point, the second coordinate of the lower right intersection point, and the slopes and intercepts of the straight lines corresponding to the four sub-boundaries, the object boundary of the target object, that is, the bounding box, is obtained. As an example, see Fig.10 , Fig.10 is a schematic diagram of an object boundary of a target object provided in an embodiment of the present application, where the object boundary is a rectangular boundary box.
[0129] In some other embodiments, the third coordinate of the intersection point between the straight line corresponding to the left sub-boundary and the straight line corresponding to the lower sub-boundary can be determined, and the fourth coordinate of the intersection point between the straight line corresponding to the right sub-boundary and the straight line corresponding to the upper sub-boundary can be determined; based on the straight lines corresponding to the sub-boundaries in each direction, combined with the third coordinate and the fourth coordinate, the object boundary of the target object in the image to be processed is obtained.
[0130] In some embodiments, after the sub-boundaries in various directions are integrated to obtain the object boundary of the target object in the image to be processed, the server can determine whether the object boundary meets the boundary abnormality condition of the target object; when the object boundary meets the boundary abnormality condition, an abnormal prompt information is output, and the abnormal prompt information is used to prompt that there is an abnormality in the target object.
[0131] In practical applications, the target object can also be detected according to the object boundary. Specifically, when the values of k and b exceed the preset range, an abnormal alarm is given, and the subsequent target object detection model is no longer input at this time, thereby improving the detection speed and recall rate. Among them, the abnormal alarm may indicate that the shape of the target object is abnormal. For example, for a rectangular target object, the k difference of the upper and lower boundaries (or left and right boundaries) exceeds the difference threshold, then the target object may have an abnormal shape problem. The abnormal alarm may also indicate that the posture of the target object is abnormal. For example, for a rectangular target object, the k of the upper and lower boundaries (or left and right boundaries) exceeds the preset interval, then the target object may have an abnormal posture, such as tilted placement.
[0132] Using the above-mentioned embodiment of the present application, firstly, a binarization process is performed on an image to be processed including a target object to obtain a binarized image; then, the binarized image is offset by an offset in at least one direction to obtain offset images in each direction; then, for the offset images in each direction, each first pixel point in the binarized image is compared with a second pixel point at a corresponding position in the offset image to obtain a comparison result, and based on the comparison result, a first pixel point in the binarized image that is different from the second pixel point in the offset image is used as a candidate boundary point of the target object; thereby, for each direction, the candidate boundary points in the direction are fitted to obtain a sub-boundary of the target object in the direction; finally, the sub-boundaries in each direction are integrated to obtain the object boundary of the target object in the image to be processed. In this way, the boundaries of objects in the image can be recognized without using a machine learning model, avoiding the introduction of model training time and inference time, and improving the recognition speed and efficiency of the boundaries of objects in the image. At the same time, it also avoids the problem of errors in model prediction results, and can improve the recognition accuracy of the boundaries of objects in the image. The object boundaries can be used to detect target objects, and can further improve the detection efficiency and accuracy of target object detection.
[0133] The following will take the object boundary of the target object as a rectangular boundary (rectangular boundary box) as an example to illustrate the exemplary application of the embodiment of the present application in an actual application scenario. First, the terms involved in the embodiment of the present application are explained, including:
[0134] 1) Region of interest (ROI): In machine vision and image processing, ROI is the area to be processed that is outlined in the image to be processed in the form of a box, circle, ellipse, irregular polygon, etc. ROI is the focus of attention in the image processing process. By defining ROI, the image processing time can be reduced and the image processing accuracy can be increased.
[0135] 2) Bounding box: In target (or object) detection, a bounding box is usually used to describe the location of the target. In some examples, the bounding box can be a rectangular box, which can be determined by the coordinates of the upper left corner vertex (x1, y1) and the coordinates of the lower right corner vertex (x2, y2) of the rectangular box.
[0136] Since the size of the input image is more sensitive when the target detection in the image is performed by machine learning, a large-size image will not only slow down the detection speed, but also cause inaccurate target detection. Therefore, the effective area of the target (i.e., ROI) can be cropped and extracted by identifying the ROI of the input image in advance, so that only the effective area of the target can be detected in the subsequent target detection process, thereby improving the target detection speed and target detection accuracy. In the related art, a machine learning model is usually used to extract the bounding box of the target (the area surrounded by the bounding box is the effective area of the target) to achieve ROI extraction. However, this solution 1) requires additional inference time of the machine learning model, which makes the ROI extraction time longer, reduces the target detection speed, and may also compress the available time for subsequent target detection, reducing the target detection accuracy; 2) the accuracy of ROI extraction may also introduce additional uncertainty (such as the model ROI prediction error, no ROI extraction, etc.), thereby losing the necessity of ROI extraction; 3) even if a simple machine learning model can be used to reduce the inference time, the effect is not good and it is basically unusable.
[0137] Based on this, an embodiment of the present application provides an image processing method to at least solve the above-mentioned problems. In the embodiment of the present application, (1) an image to be processed including a target object is binarized by an image binarization module to obtain a binarized image to achieve a rough division of the foreground and background corresponding to the target object; (2) an offset difference module is used to process the binarized image to obtain candidate boundary points of the object boundary of the target object. The number of candidate boundary points is multiple and can be recorded as a candidate boundary point set; (3) a gridding module is used to select some candidate boundary points from the candidate boundary point set in a gridding manner to convert the candidate boundary point set into a sparse candidate boundary point set, thereby improving computational efficiency; 4) Clustering the sparse candidate boundary point set through the clustering module to obtain multiple clusters (each cluster includes multiple candidate boundary points), and selecting a target cluster that meets the conditions from the multiple clusters, so that only the multiple candidate boundary points included in the target cluster are used as the boundary point set of the target object to screen out the protruding part of the target object (such as a raised corner located on the main body of the target object); (5) Performing straight line fitting on the boundary points in the boundary point set through the fitting module to obtain multiple sub-boundaries of the target object (including upper boundary, lower boundary, left boundary and right boundary). Specifically, the abnormal boundary points in the boundary point set can be screened out through the random sampling consistency (RA NSAC, Random Sample Consensus) algorithm to fit multiple straight lines as sub-boundaries; (6) Integrating the multiple sub-boundaries of the target object through the boundary integration module to obtain the object boundary of the target object in the image to be processed. The area surrounded by the object boundary is the effective area of the target object in the image to be processed (i.e., the ROI in the image to be processed).
[0138] Next, the image processing method provided by the embodiment of the present application is described in detail. Fig.11 , Fig.11 : is a flowchart of an image processing method provided in an embodiment of the present application, comprising:
[0139] Step 201 : binarizing an image to be processed including a target object through an image binarization module to obtain a binarized image.
[0140] In practical applications, since the target object (such as manufactured industrial products: display panels, wooden floors, machine parts, etc.) itself may have multiple colors and surface textures, it may have unnecessary effects on the boundary recognition of the target object. Therefore, in order to ensure the distinction between the foreground area (i.e., the target object body) and the background area (such as the area where the target object is placed) of the image to be processed, the contrast between the target object body and the background area can be increased as much as possible in the early imaging stage of the image to be processed. That is, by adjusting the lighting, the target object body is overexposed as brightly as possible, while the background area is made of light-absorbing materials as much as possible and painted black to prevent unnecessary reflections.
[0141] For the image to be processed including the target object, the image to be processed is binarized to obtain a binary image. In practical applications, (1) the channel pixel values of each pixel in the image to be processed in the three channels of R, G, and B are obtained; (2) for each pixel, the channel pixel values of the pixel in the three channels are summed and averaged to obtain the average pixel value of each pixel, so as to obtain a single-channel grayscale image of the image to be processed; (3) the pixel grayscale values of the pixels whose average pixel values are greater than the pixel value threshold are set to the maximum value (255), and the pixel grayscale values of the pixels whose average pixel values are not greater than the pixel value threshold are set to the minimum value (0), thereby converting the single-channel grayscale image into a black and white binary image to achieve image binarization. Among them, 0 represents black (darkest) and 255 represents white (brightest). See Figure 4 , Figure 4 is a schematic diagram of a binary image provided by an embodiment of the present application. Here, the binary image includes pixels with a grayscale value of 255 and pixels with a grayscale value of 0.
[0142] In practical applications, the ideal effect of image binarization is that the main body of the target object appears uniformly white, while the background area appears uniformly black. However, due to various factors such as imaging and interference, generally speaking, there will be white noise in the background area and black noise in the main body of the target object. Therefore, image binarization can be used as a rough division to distinguish the main body of the target object and the background area.
[0143] In some other embodiments, the image binarization module may also be implemented by using fixed threshold binarization and adaptive threshold binarization (such as OTSU binarization).
[0144] Step 202, through the offset difference module, the binary image is offset by one pixel in the horizontal direction and the vertical direction respectively, so as to determine multiple candidate boundary points (which can be recorded as candidate boundary point sets) of the target object in each direction based on the offset image obtained by the offset.
[0145] Exemplarily, the binary image is shifted in the horizontal direction by one pixel according to the offset direction from left to right to obtain an offset image in the horizontal direction; for each first pixel in the binary image, the pixel grayscale value of the first pixel is subtracted from the pixel grayscale value of the second pixel at the corresponding position in the offset image to obtain a grayscale value difference; the target first pixel (i.e., the difference pixel) corresponding to the grayscale value difference that is not 0 is determined as a plurality of candidate boundary points of the target object in the horizontal direction. Among them, the first pixel corresponding to the grayscale value difference of the positive value (255) is determined as the left candidate boundary point, and the first pixel corresponding to the grayscale value difference of the negative value (-255) is determined as the right candidate boundary point. See. Figure 7 , Figure 7 is a schematic diagram of candidate boundary points in the horizontal direction provided by an embodiment of the present application. Here, the candidate boundary points in the horizontal direction include left candidate boundary points and right candidate boundary points.
[0146] Exemplarily, the binary image is offset by one pixel in the vertical direction according to the offset direction from top to bottom to obtain an offset image in the vertical direction; for each first pixel in the binary image, the pixel grayscale value of the first pixel is subtracted from the pixel grayscale value of the second pixel at the corresponding position in the offset image to obtain a grayscale value difference; the target first pixel (i.e., the difference pixel) corresponding to the grayscale value difference that is not 0 is determined as a plurality of candidate boundary points of the target object in the vertical direction. Among them, the first pixel corresponding to the grayscale value difference of the positive value (255) is determined as the upper candidate boundary point, and the first pixel corresponding to the grayscale value difference of the negative value (-255) is determined as the lower candidate boundary point. See. Figure 8 , Figure 8 is a schematic diagram of candidate boundary points in the vertical direction provided by an embodiment of the present application. Here, the candidate boundary points in the vertical direction include upper candidate boundary points and lower candidate boundary points.
[0147] In actual applications, the candidate boundary points obtained in step 202 include: boundary points of the target object body, noise boundary points inside the target object body (i.e., black noise points inside the target object body displayed in the binary image), and noise boundary points outside the target object body (i.e., background area) (i.e., white noise points outside the target object body displayed in the binary image).
[0148] In some other embodiments, the offset subtraction module may also be implemented using a traditional edge detection algorithm (such as Canny edge detection).
[0149] Step 203: transform the candidate boundary point set into a sparse candidate boundary point set through a gridding module.
[0150] For the binary image with the determined candidate boundary point set, start from the first row of pixels and scan each pixel from left to right in the current row. When the target candidate boundary point corresponding to the positive value (255) is scanned, save the coordinates of the target candidate boundary point in the temporary storage area, and jump from the current row to the target row (such as jumping from the first row to the eleventh row) with the target step value (such as step = 10), and enter the next scan for the target row. And so on, until the last row is scanned, and all the target candidate boundary points obtained by scanning are used as the left sparse candidate boundary points in the sparse candidate boundary point set. See Figure 6 , Figure 6 is a schematic diagram of the left candidate boundary point in the horizontal direction provided by an embodiment of the present application.
[0151] For the binary image with the determined candidate boundary point set, start from the first row of pixels, scan each pixel from right to left in the current row, and when the target candidate boundary point corresponding to the negative value (-255) is scanned, save the coordinates of the target candidate boundary point in the temporary storage area, and jump from the current row to the target row (such as jumping from the first row to the eleventh row) with the target step value (such as step = 10), and enter the next scan for the target row. And so on, until the last row is scanned, and all the target candidate boundary points obtained by scanning are used as the right sparse candidate boundary points in the sparse candidate boundary point set.
[0152] For the binary image with the determined candidate boundary point set, starting from the first column of pixels, scan each pixel from top to bottom in the current column. When the target candidate boundary point corresponding to the positive value (255) is scanned, the coordinates of the target candidate boundary point are saved in the temporary storage area, and jump from the current column to the target column (such as jumping from the first column to the eleventh column) with the target step value (such as step=10), and enter the next scan for the target column. And so on, until the last column is scanned, and all the target candidate boundary points obtained by scanning are used as the upper sparse candidate boundary points in the sparse candidate boundary point set.
[0153] For the binary image with the determined candidate boundary point set, start from the first column of pixels, scan each pixel from bottom to top in the current column, and when the target candidate boundary point corresponding to the negative value (-255) is scanned, save the coordinates of the target candidate boundary point in the temporary storage area, and jump from the current column to the target column (such as jumping from the first column to the eleventh column) with the target step value (such as step = 10), and enter the next scan for the target column. And so on, until the last column is scanned, and all the target candidate boundary points obtained by scanning are used as the lower sparse candidate boundary points in the sparse candidate boundary point set.
[0154] In this way, the amount of calculation can be reduced. The larger the step, the faster the calculation speed, but the accuracy of ROI positioning will be affected. You can select the appropriate step based on experience.
[0155] In some other embodiments, the gridding module may also be implemented through a strategy of random sampling point sets.
[0156] Step 204, clustering the sparse candidate boundary point set through a clustering module to obtain multiple clusters (each cluster includes multiple candidate boundary points), and selecting a target cluster that meets the conditions, and using the multiple candidate boundary points included in the target cluster as the boundary point set of the target object.
[0157] Here, the target object may not be an ideal rectangle, and may include a normal protruding part (belonging to the target object itself, such as a raised corner located on the main body of the target object). To prevent the boundary of the protruding part of the target object from affecting the positioning of the main body of the target object, the candidate boundary points of the normal protruding part can be filtered out, leaving only the candidate boundary points of the main body of the target object.
[0158] In practical applications, the sparse candidate boundary point set includes a left sparse candidate boundary point set, a right sparse candidate boundary point set, an upper sparse candidate boundary point set, and a lower sparse candidate boundary point set. It should be noted that step 204 is to process the four sparse candidate boundary point sets of the upper, lower, left and right included in the sparse candidate boundary point set respectively. Specifically, for each sparse candidate boundary point set: the sparse candidate boundary point set is clustered to obtain multiple clusters, each cluster includes multiple candidate boundary points, and then a target cluster that meets the conditions is selected from the multiple clusters, and the candidate boundary points included in other clusters except the target cluster in the multiple clusters are filtered out to filter out the candidate boundary points of the protruding part of the target object (such as a raised corner on the main body of the target object).
[0159] Exemplarily, the clustering process can be implemented using a K-means clustering algorithm: using the K-means clustering algorithm requires specifying in advance the number m of clusters obtained by clustering, and the number m can be determined by the shape of the target object, for example, m=2n+1 (n is the number of protruding parts of the target object), that is, if there is only one protruding part, m can be set to 3, and if there are two protruding parts, m can be set to 5. When selecting a target cluster that meets the conditions from multiple clusters, 1) (n+1) clusters with closer inter-cluster distances (such as (n+1) clusters with inter-cluster distances below a distance threshold, and (n+1) clusters with higher inter-cluster distances in ascending order) can be selected from the multiple clusters as target clusters. For example, if there is only one protruding part (n=1), m can be set to 3, and after clustering, two clusters with similar distances between clusters can be selected as target clusters; if there are two protruding parts (n=2), m can be set to 5, and after clustering, three clusters with similar distances between clusters can be selected as target clusters. 2) It is also possible to select from multiple clusters the number of boundary points of candidate boundary points that meets the quantity condition (such as several clusters whose number of boundary points is greater than the quantity threshold, or several clusters whose number of boundary points is ranked first in ascending order) as target clusters.
[0160] As an example, see Figure 5 , Figure 5 is a schematic diagram of the first candidate boundary point on the left provided in an embodiment of the present application. Here, Figure 5 As shown in (1), the clusters obtained by clustering the candidate boundary points on the left include cluster 1, cluster 2, cluster 3 and cluster 4. Figure 5 As shown in (2), target clusters that meet the conditions are selected from multiple clusters, namely cluster 1 and cluster 3.
[0161] In practical applications, step 204 is optional. For example, when the number of outliers in the sparse candidate boundary point set is less than the threshold number of outliers, step 204 can be omitted. An outlier is a candidate boundary point that is more than a distance threshold (such as 10 pixels) from the boundary of the target object.
[0162] In some other embodiments, the clustering module can be implemented using various clustering algorithms (such as hierarchical clustering, K-Means, DBSCAN, etc.).
[0163] Step 205 , performing straight line fitting on the boundary points in the boundary point set through a fitting module to obtain multiple sub-boundaries (including an upper boundary, a lower boundary, a left boundary, and a right boundary) of the target object.
[0164] In practical applications, the RANSAC algorithm can be used to filter out abnormal boundary points in the boundary point set to fit multiple straight lines as sub-boundaries. The RANSAC algorithm uses an iterative method to estimate the parameters of the mathematical model from a set of observed data containing outliers. It is a robust algorithm that only uses a normal point set to calculate the mathematical model and can exclude the interference of outliers. The RANSAC algorithm contains two important parameters: the points sampled when generating the mathematical model in each round of iteration (e) and the number of iterations (i).
[0165] In practical applications, the boundary point set also includes an upper boundary point set, a lower boundary point set, a left boundary point set and a right boundary point set. Step 205 also processes the four boundary point sets of the upper, lower, left and right boundaries included in the boundary point set separately. Specifically, for each boundary point set: 1) randomly sample multiple boundary points from the boundary point set as a fitting point set; 2) perform straight line fitting on the multiple boundary points included in the fitting point set to obtain a straight line mathematical model (y=kx+b); 3) calculate the distance from each boundary point in the boundary point set to the straight line, and take the boundary points with a distance lower than the distance threshold as normal boundary points, and record the number of normal boundary points; 4) repeat the above steps 1)-3) until the iteration ends; 5) compare the number of normal boundary points in each iteration, and use the straight line mathematical model fitted when the normal boundary points are the largest as the mathematical model corresponding to the sub-boundary of the target object, so as to calculate the object boundary based on the mathematical model. See Fig. 9 , Fig. 9 It is a schematic diagram of the straight line corresponding to the left sub-boundary provided in an embodiment of the present application.
[0166] In some other embodiments, the fitting module may also be implemented by simple linear regression or regression using a neural network.
[0167] Step 206 , integrating multiple sub-boundaries of the target object through a boundary integration module to obtain an object boundary of the target object in the image to be processed.
[0168] After calculating the mathematical models (y=kx+b) corresponding to the multiple sub-boundaries of the target object (including the upper boundary, lower boundary, left boundary and right boundary), the coordinates of the upper left intersection point and the lower right intersection point are calculated and output through the intersection of the two lines, and the slopes and intercepts of the four sub-boundaries are output at the same time, that is, k and b in y=kx+b. Based on the output coordinates of the upper left intersection point, the coordinates of the lower right intersection point, and the slopes and intercepts of the lines corresponding to the four sub-boundaries, the object boundary of the target object, that is, the bounding box, is obtained. As an example, see Fig.10 , Fig.10 is a schematic diagram of an object boundary of a target object provided in an embodiment of the present application, where the object boundary is a rectangular boundary box.
[0169] In addition, in step 206, the target object can also be detected according to the object boundary. Specifically, when the values of k and b exceed the preset range, an abnormal alarm is given, and at this time, it is no longer input into the subsequent target object detection model, thereby improving the detection speed and recall rate. Among them, the abnormal alarm may indicate that the shape of the target object is abnormal. For example, for a rectangular target object, the k difference of the upper and lower boundaries (or left and right boundaries) exceeds the difference threshold, then the target object may have an abnormal shape problem. The abnormal alarm may also indicate that the posture of the target object is abnormal. For example, for a rectangular target object, the k of the upper and lower boundaries (or left and right boundaries) exceeds the preset interval, then the target object may have an abnormal posture, such as tilted placement.
[0170] Next, the application scenario of the image processing method provided in the embodiment of the present application is described. In industrial manufacturing scenarios, emerging technologies such as machine learning are often used to perform quality inspection on the appearance of products (i.e., the above-mentioned target objects) to ensure product consistency, yield rate, and safety, and to achieve full automation of production line quality inspection. In order to identify more detailed appearance defect problems of products, high-resolution images are often generated in the imaging stage as input to the downstream appearance defect detection model. Since machine learning is used to detect targets in images, it is sensitive to the size of the input image, and large-size images will not only slow down the detection speed, but also cause inaccurate target detection. Therefore, the effective area of the product (i.e., ROI) can be cropped and extracted by identifying the ROI of the input image in advance, so that only the effective area of the product can be detected in the subsequent product appearance defect detection process, thereby improving the product detection speed and product detection accuracy.
[0171] Based on this, the scenario of accurate extraction of the product's bounding box (the area enclosed by the bounding box is the effective area of the product) is mainly the pre-preprocessing of downstream tasks (i.e., product appearance defect detection tasks), and its application scenarios may include: 1) Product body extraction: extract the product to be processed in the entire image, filter out the background or other components, and send the main part of the product to the downstream task for further processing. 2) Target alignment: In the scenario of clamping by hardware (such as a product gripper), the default bounding box of the product can be preset for the hardware machine, and then in the actual process of clamping the target product, the target bounding box of the target product can be extracted. When the target bounding box is offset from the default bounding box, the hardware can be automatically fine-tuned through the hardware machine to clamp the target product through the adjusted hardware. In this way, the bounding box of the target can be aligned on each different product, which is convenient for subsequent hardware clamping and other operations. 3) Coarse defect classification: After obtaining the boundary box of the product, it can be cross-compared with the parameters of the normal sample of the product. If the length, slope and other parameters deviate too much from the normal sample or are inconsistent, the product can be judged as a problem product, and no downstream tasks are required, and it can be directly treated as a non-conforming product. It should be noted that the product can be a) LCD panel, OLED panel; b) ceramic tile, wooden floor; c) battery cell; d) square box, square bottle; e) license plate, house number; f) rectangular parts; etc.
[0172] By applying the above embodiments of the present application, 1) the accuracy and speed of ROI recognition are improved; 2) the structure is clear, each processing module has good generalization ability and strong feasibility; 3) the reasoning speed is much lower than the operation speed of the deep algorithm in the GPU, which saves time and reserves a huge amount of time for the optimization of the subsequent defect detection module; 4) due to the small number of parameters and efficient, stable and controllable operation, it can not only adapt to a variety of machines and manufacturing scenarios, but also save more time for downstream detection models to optimize the detection accuracy, thereby greatly improving the usability in industrial intelligent manufacturing; 5) the generated boundary is stable and accurate, the algorithm logic is clear, stable and controllable, and the calculation results of each step can be visualized, which is convenient for quickly locating problems when the algorithm works abnormally; 6) the generated boundary and the real object have high fitting accuracy and can be used for margin measurement work such as length and slope.
[0173] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0174] The following further describes an exemplary structure of the image processing device 553 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2As shown, the software modules stored in the image processing device 553 of the memory 550 may include: a binarization module 5531, which is used to perform binarization processing on the image to be processed including the target object to obtain a binarized image; an offset module 5532, which is used to perform an offset processing on the binarized image in at least one direction by an offset amount to obtain an offset image in each of the directions; a comparison module 5533, which is used to compare each first pixel point in the binarized image with a second pixel point at a corresponding position in the offset image for each of the directions to obtain a comparison result, and based on the comparison result, use the first pixel point in the binarized image that is different from the second pixel point in the offset image as a candidate boundary point of the target object; a fitting module 5534, which is used to fit the candidate boundary points in each of the directions to obtain a sub-boundary of the target object in the direction; and an integration module 5535, which is used to integrate the sub-boundaries in each of the directions to obtain an object boundary of the target object in the image to be processed, and the object boundary is used for detecting the target object.
[0175] In some embodiments, the binarization module 5531 is also used to obtain the pixel value of each pixel in the image to be processed; the pixel whose pixel value exceeds the pixel threshold is used as the third pixel, and the pixel whose pixel value does not exceed the pixel threshold is used as the fourth pixel; the pixel grayscale value of the third pixel is determined as the first grayscale value, and the pixel grayscale value of the fourth pixel is determined as the second grayscale value, so as to obtain the binary image.
[0176] In some embodiments, the binarization module 5531 is also used to perform the following processing on each pixel point in the image to be processed: obtain the channel pixel value of the pixel point on the red R, green G, and blue B color channels respectively; average the channel pixel values on multiple color channels to obtain an average processing result, and use the average processing result as the pixel value of the pixel point.
[0177] In some embodiments, the offset module 5532 is also used to offset the binary image by one pixel in the horizontal direction according to a first offset direction to obtain the offset image in the horizontal direction; and to offset the binary image by one pixel in the vertical direction according to a second offset direction to obtain the offset image in the vertical direction.
[0178] In some embodiments, the comparison module 5533 is also used to perform the following processing for each of the first pixel points in the binary image: determine the pixel value difference between the pixel value of the first pixel point and the pixel value of the second pixel point at the corresponding position in the offset image; when the pixel value difference is not 0, obtain a comparison result that there is a difference between the first pixel point and the second pixel point at the corresponding position in the offset image.
[0179] In some embodiments, the comparison module 5533 is further used to perform an offset processing on the binary image in the horizontal direction according to the offset direction from left to right, so as to obtain the offset image in the horizontal direction; the comparison module 5533 is further used to use the first pixel point whose pixel value difference is greater than 0 as the left candidate boundary point of the target object, and use the first pixel point whose pixel value difference is less than 0 as the right candidate boundary point of the target object; the fitting module 5534 is further used to fit the left candidate boundary point in the horizontal direction to obtain the left sub-boundary of the target object in the horizontal direction, and to fit the right candidate boundary point in the horizontal direction to obtain the right sub-boundary of the target object in the horizontal direction.
[0180] In some embodiments, the comparison module 5533 is further used to perform an offset processing on the binary image in the vertical direction by an offset amount according to the offset direction from top to bottom, so as to obtain the offset image in the vertical direction; the comparison module 5533 is further used to use the first pixel point whose pixel value difference is greater than 0 as the upper candidate boundary point of the target object, and use the first pixel point whose pixel value difference is less than 0 as the lower candidate boundary point of the target object; the fitting module 5534 is further used to fit the upper candidate boundary point in the vertical direction to obtain the upper sub-boundary of the target object in the vertical direction, and to fit the lower candidate boundary point in the vertical direction to obtain the lower sub-boundary of the target object in the vertical direction.
[0181] In some embodiments, when there are multiple candidate boundary points, the comparison module 5533 is also used to cluster the multiple candidate boundary points to obtain at least one cluster, each cluster including at least one candidate boundary point; a target cluster that meets the conditions is selected from the at least one cluster, and the candidate boundary points included in the target cluster are used as first candidate boundary points; the fitting module 5534 is also used to fit the first candidate boundary points in each direction to obtain the sub-boundary of the target object in the direction.
[0182] In some embodiments, the comparison module 5533 is also used to obtain the number of boundary points of the candidate boundary points included in each of the clustering clusters, and use the clustering cluster whose number of boundary points meets the quantity condition as the target clustering cluster; or, obtain the inter-cluster distance between any two clustering clusters in the at least one clustering cluster, and use the clustering cluster whose inter-cluster distance meets the distance condition as the target clustering cluster.
[0183] In some embodiments, when the number of the candidate boundary points is multiple and the direction is a horizontal direction, the comparison module 5533 is also used to perform the following processing starting from the first pixel point of the first row of the binary image: for the current row to be scanned, according to the first scanning direction, starting from the first first pixel point of the current row, the first pixel point of the current row is scanned in sequence; when one of the multiple candidate boundary points is scanned, the scanned candidate boundary point is used as the candidate boundary point in the horizontal direction; determine whether there is a target row that is a preset row step value away from the current row; if the target row exists, jump from the current row to the target row to use the target row as the current row to be scanned, and execute the above operations in a loop; if the target row does not exist, execute the end operation.
[0184] In some embodiments, when the number of the candidate boundary points is multiple and the direction is a vertical direction, the comparison module 5533 is also used to perform the following processing starting from the first pixel point of the first column of the binary image: for the current column to be scanned, according to the second scanning direction, starting from the first first pixel point of the current column, the first pixel points of the current column are scanned in sequence; when one candidate boundary point among the multiple candidate boundary points is scanned, the scanned candidate boundary point is used as the candidate boundary point in the vertical direction; determine whether there is a target column that is a preset column step value away from the current column; if the target column exists, jump from the current column to the target column to use the target column as the current column to be scanned, and execute the above operations in a loop; if the target column does not exist, execute the end operation.
[0185] In some embodiments, there are multiple candidate boundary points in the direction, and the fitting module 5534 is further used to sample multiple second candidate boundary points from the multiple candidate boundary points in the direction, and perform straight line fitting on the multiple second candidate boundary points to obtain corresponding straight lines; determine the distance between each of the candidate boundary points and the straight line, and determine the number of boundary points of the candidate boundary points whose distance is lower than the distance threshold; loop through the above operations until the iterative process ends; determine the maximum number of boundary points corresponding to multiple iterative processes, and determine the target straight line obtained by the target iterative process corresponding to the maximum number of boundary points; determine the target straight line as the sub-boundary of the target object in the direction.
[0186] In some embodiments, when the sub-boundaries in the direction include a left sub-boundary, a right sub-boundary, an upper sub-boundary and a lower sub-boundary, the integration module 5535 is also used to determine the first coordinates of the intersection of the straight line corresponding to the left sub-boundary and the straight line corresponding to the upper sub-boundary, and to determine the second coordinates of the intersection of the straight line corresponding to the right sub-boundary and the straight line corresponding to the lower sub-boundary; based on the straight lines corresponding to the sub-boundaries in each of the directions, combined with the first coordinates and the second coordinates, the object boundary of the target object in the image to be processed is obtained.
[0187] In some embodiments, the integration module 5535 is also used to determine whether the object boundary meets the boundary exception condition of the target object; when the object boundary meets the boundary exception condition, output exception prompt information, and the exception prompt information is used to prompt that there is an exception with the target object.
[0188] Using the above-mentioned embodiment of the present application, firstly, a binarization process is performed on an image to be processed including a target object to obtain a binarized image; then, the binarized image is offset by an offset in at least one direction to obtain offset images in each direction; then, for the offset images in each direction, each first pixel point in the binarized image is compared with a second pixel point at a corresponding position in the offset image to obtain a comparison result, and based on the comparison result, a first pixel point in the binarized image that is different from the second pixel point in the offset image is used as a candidate boundary point of the target object; thereby, for each direction, the candidate boundary points in the direction are fitted to obtain a sub-boundary of the target object in the direction; finally, the sub-boundaries in each direction are integrated to obtain the object boundary of the target object in the image to be processed. In this way, the boundaries of objects in the image can be recognized without using a machine learning model, avoiding the introduction of model training time and inference time, and improving the recognition speed and efficiency of the boundaries of objects in the image. At the same time, it also avoids the problem of errors in model prediction results, and can improve the recognition accuracy of the boundaries of objects in the image. The object boundaries can be used to detect target objects, and can further improve the detection efficiency and accuracy of target object detection.
[0189] The embodiment of the present application also provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the image processing method provided in the embodiment of the present application.
[0190] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the image processing method provided by the embodiment of the present application.
[0191] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
[0192] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0193] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).
[0194] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.
[0195] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. An image processing method, It is characterized in that The method comprises: Binarization is performed on the image to be processed including the target object to obtain a binary image; Performing an offset process on the binary image in multiple directions with an offset amount to obtain offset images in each of the directions, wherein the multiple directions include a first offset direction and a second offset direction, the first offset direction is a horizontal direction, and the second offset direction is a vertical direction; For the offset image in each of the directions, each first pixel point in the binary image is compared with a second pixel point at a corresponding position in the offset image to obtain a comparison result, and Based on the comparison result, taking a first pixel point in the binary image that is different from a second pixel point in the offset image as a candidate boundary point of the target object; For each of the directions, fitting the candidate boundary points in the direction to obtain a sub-boundary of the target object in the direction; The sub-boundaries in each of the directions are integrated to obtain an object boundary of the target object in the image to be processed, and the object boundary is used for detecting the target object.
2. The method according to claim 1, It is characterized in that The step of performing binarization processing on the image to be processed including the target object to obtain a binarized image includes: Obtaining the pixel value of each pixel in the image to be processed; The pixel point whose pixel value exceeds the pixel threshold is used as the third pixel point, and the pixel point whose pixel value does not exceed the pixel threshold is used as the fourth pixel point; The pixel grayscale value of the third pixel point is determined as the first grayscale value, and the pixel grayscale value of the fourth pixel point is determined as the second grayscale value, so as to obtain the binary image.
3. The method according to claim 2, It is characterized in that The step of obtaining the pixel value of each pixel in the image to be processed includes: For each pixel in the image to be processed, the following processing is performed respectively: Obtain channel pixel values of the pixel point on the red R, green G, and blue B color channels respectively; The channel pixel values on the multiple color channels are averaged to obtain an average processing result, and the average processing result is used as the pixel value of the pixel point.
4. The method according to claim 1, It is characterized in that The step of performing an offset process on the binary image in multiple directions by an offset amount to obtain an offset image in each of the directions includes: According to the first offset direction, the binary image is offset by one pixel in a horizontal direction to obtain an offset image in the horizontal direction; And, according to the second offset direction, the binary image is offset by one pixel in the vertical direction to obtain an offset image in the vertical direction.
5. The method according to claim 1, It is characterized in that The step of comparing each first pixel point in the binary image with a second pixel point at a corresponding position in the offset image to obtain a comparison result includes: For each of the first pixels in the binary image, the following processing is performed respectively: Determining a pixel value difference between a pixel value of a first pixel point and a pixel value of a second pixel point at a corresponding position in the offset image; When the pixel value difference is not 0, a comparison result is obtained indicating that there is a difference between the first pixel point and a second pixel point at a corresponding position in the offset image.
6. The method according to claim 5, It is characterized in that The step of performing an offset process on the binary image in multiple directions by an offset amount to obtain an offset image in each of the directions includes: According to the offset direction from left to right, the binary image is subjected to an offset process in the horizontal direction by an offset amount to obtain an offset image in the horizontal direction; The step of taking, based on the comparison result, a first pixel point in the binary image that is different from a second pixel point in the offset image as a candidate boundary point of the target object includes: The first pixel point whose pixel value difference is greater than 0 is used as a candidate boundary point on the left side of the target object, and the first pixel point whose pixel value difference is less than 0 is used as a candidate boundary point on the right side of the target object; The step of fitting the candidate boundary points in the direction to obtain the sub-boundary of the target object in the direction includes: The left candidate boundary points in the horizontal direction are fitted to obtain the left subboundary of the target object in the horizontal direction, and the right candidate boundary points in the horizontal direction are fitted to obtain the right subboundary of the target object in the horizontal direction.
7. The method according to claim 5, It is characterized in that The step of performing an offset process on the binary image in multiple directions by an offset amount to obtain an offset image in each of the directions includes: According to the offset direction from top to bottom, the binary image is subjected to an offset process in the vertical direction by an offset amount to obtain an offset image in the vertical direction; The step of taking, based on the comparison result, a first pixel point in the binary image that is different from a second pixel point in the offset image as a candidate boundary point of the target object includes: The first pixel point whose pixel value difference is greater than 0 is used as an upper candidate boundary point of the target object, and the first pixel point whose pixel value difference is less than 0 is used as a lower candidate boundary point of the target object; The step of fitting the candidate boundary points in the direction to obtain the sub-boundary of the target object in the direction includes: The upper candidate boundary points in the vertical direction are fitted to obtain an upper subboundary of the target object in the vertical direction, and the lower candidate boundary points in the vertical direction are fitted to obtain a lower subboundary of the target object in the vertical direction.
8. The method according to claim 1, It is characterized in that When there are multiple candidate boundary points, after taking, based on the comparison result, a first pixel point in the binarized image that is different from a second pixel point in the offset image as a candidate boundary point of the target object, the method further includes: Performing clustering processing on the plurality of candidate boundary points to obtain at least one cluster, each of the clusters including at least one candidate boundary point; Selecting a target cluster that meets the condition from the at least one cluster, and using the candidate boundary point included in the target cluster as the first candidate boundary point; The step of fitting the candidate boundary points in each direction to obtain the sub-boundary of the target object in the direction includes: For each of the directions, the first candidate boundary points in the direction are fitted to obtain a sub-boundary of the target object in the direction.
9. The method according to claim 1, It is characterized in that When there are multiple candidate boundary points and the direction is a horizontal direction, after taking a first pixel point in the binary image that is different from a second pixel point in the offset image as a candidate boundary point of the target object based on the comparison result, the method further includes: Starting from the first pixel of the first row of the binary image, the following processing is performed: For a current row to be scanned, starting from a first pixel point of the current row, the first pixel points of the current row are scanned in sequence according to a first scanning direction; When one of the plurality of candidate boundary points is scanned, the scanned candidate boundary point is used as a candidate boundary point in the horizontal direction; Determine whether a target row with a preset row step value away from the current row exists; If the target row exists, jump from the current row to the target row, so as to use the target row as the current row to be scanned, and execute the above operation in a loop; If the target row does not exist, the end operation is performed.
10. The method according to claim 1, It is characterized in that When there are multiple candidate boundary points and the direction is a vertical direction, after taking a first pixel point in the binarized image that is different from a second pixel point in the offset image as a candidate boundary point of the target object based on the comparison result, the method further includes: Starting from the first pixel in the first column of the binary image, the following processing is performed: For a current column to be scanned, starting from the first pixel point of the current column, the first pixel points of the current column are scanned sequentially according to a second scanning direction; When one of the candidate boundary points is scanned, the scanned candidate boundary point is used as a candidate boundary point in the vertical direction; Determine whether there is a target column whose distance from the current column is a preset column step value; If the target column exists, jump from the current column to the target column to use the target column as the current column to be scanned, and execute the above operation in a loop; If the target column does not exist, the end operation is performed.
11. The method according to claim 1, It is characterized in that There are multiple candidate boundary points in the direction, and fitting the candidate boundary points in the direction to obtain the sub-boundary of the target object in the direction includes: Sampling a plurality of second candidate boundary points from a plurality of candidate boundary points in the direction, and performing straight line fitting on the plurality of second candidate boundary points to obtain corresponding straight lines; Determine the distance between each of the candidate boundary points and the straight line, and determine the number of boundary points of the candidate boundary points whose distance is lower than a distance threshold; The above operation is executed repeatedly until the iteration process ends; Determine the maximum number of boundary points among the numbers of boundary points corresponding to the multiple iteration processes, and determine the target straight line obtained by the target iteration process corresponding to the maximum number of boundary points; The target straight line is determined as a sub-boundary of the target object in the direction.
12. An image processing device, It is characterized in that The device comprises: A binarization module, used for performing binarization processing on the image to be processed including the target object to obtain a binarized image; An offset module is used to perform an offset process on the binary image in multiple directions by an offset amount to obtain an offset image in each of the directions, wherein the multiple directions include a first offset direction and a second offset direction, wherein the first offset direction is a horizontal direction and the second offset direction is a vertical direction; a comparison module, for comparing, for the offset images in each of the directions, each first pixel point in the binary image with a second pixel point at a corresponding position in the offset image to obtain a comparison result, and based on the comparison result, taking a first pixel point in the binary image that is different from a second pixel point in the offset image as a candidate boundary point of the target object; A fitting module, used for fitting the candidate boundary points in each direction to obtain a sub-boundary of the target object in the direction; The integration module is used to integrate the sub-boundaries in each of the directions to obtain the object boundary of the target object in the image to be processed, and the object boundary is used to detect the target object.
13. An electronic device, It is characterized in that The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 11 when executing the computer executable instructions stored in the memory.
14. A computer-readable storage medium storing computer-executable instructions, It is characterized in that When the computer executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 11 is implemented.
15. A computer program product comprising a computer program or computer executable instructions, It is characterized in that When the computer program or computer executable instruction is executed by a processor, the image processing method according to any one of claims 1 to 11 is implemented.
