Image processing method and device for 5S management

Abstract

The embodiment of the invention provides an image processing method and device for 5S management. The image processing method for 5S management comprises the following steps: acquiring a to-be-processed image shot for a target area; identifying and positioning at least one target object from the to-be-processed image by using a target detection technology; extracting local features of each target object from the image region where each target object is located; determining the satisfaction degree between each target object and the corresponding 5S management evaluation rule according to the local features of each target object; summarizing the obtained satisfaction degrees to generate a first score of the object hierarchy; and generating a 5S management evaluation score of the to-be-processed image based on the first score.

Description

Technical Field

[0001] The embodiments of the present specification generally relate to the field of computer technology, and more particularly to an image processing method and apparatus for 5S management. Background Art

[0002] At present, image processing technology has been widely used in various fields. For production-oriented enterprises such as chemical enterprises, 5S management, as a widely recognized on-site management method, includes five aspects: sorting (Seiri), rectifying (Seiton), cleaning (Seiso), cleaning (Seiketsu) and quality (Shitsuke), which is crucial to ensure production safety, improve production efficiency and ensure product quality.

[0003] The traditional 5S management method mainly relies on manual inspection and evaluation, which has many limitations. For example, manual inspection is inefficient and prone to omissions and inaccurate subjective judgments; it is difficult to achieve comprehensive and timely monitoring of large-scale chemical production areas. Therefore, it is of great practical significance to provide an image processing technology to improve the efficiency of 5S management and reduce the cost of management in a targeted manner. Summary of the invention

[0004] In view of the above, the embodiments of this specification provide an image processing method and device for 5S management. Using this image processing scheme, the first score of the object level is obtained by the satisfaction between the extracted local features and the corresponding 5S management evaluation rules, so as to provide a reference for the final 5S management evaluation score. This scheme can analyze the degree of compliance between the image to be processed and the corresponding 5S management evaluation rules from the perspective of image understanding, so that it can be suitable for scenes without corresponding standard images, and can also greatly reduce the storage volume of standard images.

[0005] According to one aspect of an embodiment of the present specification, there is provided an image processing method for 5S management, comprising: acquiring an image to be processed taken for a target area; identifying and locating at least one target object from the image to be processed using target detection technology; extracting local features of each target object from the image area where each target object is located; determining, based on the local features of each target object, a degree of satisfaction between each target object and a corresponding 5S management evaluation rule; summarizing the obtained degrees of satisfaction to generate a first score at the object level; and generating a 5S management evaluation score for the image to be processed based on the first score.

[0006] According to another aspect of an embodiment of the present specification, there is provided an image processing device for 5S management, comprising: an image acquisition unit, configured to acquire an image to be processed taken for a target area; a target detection unit, configured to identify and locate at least one target object from the image to be processed using a target detection technology; a feature extraction unit, configured to extract local features of each target object from the image area where each target object is located; a first score determination unit, configured to determine the degree of satisfaction between each target object and a corresponding 5S management evaluation rule based on the local features of each target object; summarizing the obtained degrees of satisfaction to generate a first score of the object level; and a final score generation unit, configured to generate a 5S management evaluation score of the image to be processed based on the first score.

[0007] According to another aspect of an embodiment of the present specification, there is provided an image processing device for 5S management, comprising: at least one processor, and a memory coupled to the at least one processor, the memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the image processing method for 5S management as described above.

[0008] According to another aspect of the embodiments of the present specification, there is provided a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the image processing method for 5S management as described above is implemented.

[0009] According to another aspect of an embodiment of the present specification, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to implement the image processing method for 5S management as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A further understanding of the nature and advantages of the content of this specification may be achieved by referring to the following drawings. In the drawings, similar components or features may have the same reference numerals.

[0011] Figure 1 An exemplary architecture of an image processing method and apparatus for 5S management according to an embodiment of the present specification is shown.

[0012] Figure 2 A flowchart showing an example of an image processing method for 5S management according to an embodiment of the present specification.

[0013] Figure 3 A schematic diagram showing an example of a process of generating a 5S management evaluation rule according to an embodiment of the present specification.

[0014] Figure 4A flowchart showing still another example of a process for generating a 5S management evaluation rule according to an embodiment of the present specification.

[0015] Figure 5 A flowchart showing still another example of the image processing method for 5S management according to an embodiment of the present specification.

[0016] Figure 6 A flowchart showing an example of a process for determining a global similarity between an image to be processed and a corresponding reference image according to an embodiment of the present specification.

[0017] Figure 7 A flowchart showing one example of a generation process of a 5S management evaluation score according to an embodiment of the present specification.

[0018] Figure 8 A block diagram showing an example of an image processing apparatus for 5S management according to an embodiment of the present specification.

[0019] Fig. 9 A schematic diagram showing an example of an image processing apparatus for 5S management according to an embodiment of the present specification. DETAILED DESCRIPTION

[0020] The subject matter described herein will be discussed below with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the embodiments of this specification. Various processes or components may be omitted, substituted or added to each example as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0021] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0022] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0023] In view of this, the embodiment of this specification proposes an image processing scheme for 5S management. In the image processing scheme for 5S management, the first score of the object level is obtained by the satisfaction between the extracted local features and the corresponding 5S management evaluation rules, so as to provide a reference for the final 5S management evaluation score. This scheme can analyze the degree of compliance between the image to be processed and the corresponding 5S management evaluation rules from the perspective of image understanding, so that it can be suitable for scenes without corresponding standard images, and can also greatly reduce the storage volume of standard images.

[0024] The image processing method and apparatus for 5S management according to the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 An exemplary architecture 100 of an image processing method and apparatus for 5S management according to an embodiment of the present specification is shown.

[0026] exist Figure 1 In the embodiment, the network 110 is used to interconnect the terminal device 120 and the application server 130 .

[0027] The network 110 may be any type of network capable of interconnecting network entities. The network 110 may be a single network or a combination of various networks. In terms of coverage, the network 110 may be a local area network (LAN), a wide area network (WAN), etc. In terms of carrier media, the network 110 may be a wired network, a wireless network, etc. In terms of data exchange technology, the network 110 may be a circuit switching network, a packet switching network, etc.

[0028] The terminal device 120 may be any type of electronic computing device that can connect to the network 110, access a server or website on the network 110, process data or signals, etc. For example, the terminal device 120 may be a laptop, a tablet computer, a smart phone, a camera, an automatic inspection robot with a camera, etc. Figure 1 Only one terminal device is shown in FIG. 1 , but it should be understood that a different number of terminal devices may be connected to the network 110 .

[0029] In one embodiment, the terminal device 120 may be used by a user. In some examples, the terminal device 120 may transmit the image to be processed taken by the user to the application server 130. In one embodiment, the terminal device 120 may be automatically operated. In some examples, the terminal device 120 may transmit the image to be processed taken to the application server 130. In some cases, the terminal device 120 may also receive a relevant response (e.g., a 5S management evaluation score of the image to be processed) from the application server 130.

[0030] However, it should be understood that, in one embodiment, the application server 130 may execute the above-mentioned image processing method for 5S management. In one embodiment, the terminal device 120 may also execute the above-mentioned image processing method for 5S management instead of interacting with the application server 130.

[0031] It should be understood that Figure 1 All network entities shown in the figure are exemplary. Any other network entities may be involved in the architecture 100 according to specific application requirements.

[0032] Figure 2 A flowchart showing an example of an image processing method 200 for 5S management according to an embodiment of the present specification.

[0033] like Figure 2 As shown, at 210, an image to be processed shot for a target area is acquired.

[0034] In this embodiment, the image to be processed may be a photo taken by a user through a terminal device (such as a smart phone). In this embodiment, the image to be processed may also be a photo taken by an automatic device (such as a surveillance camera or a patrol robot, etc.). In some examples, individual workstations in an office area, production equipment in a production workshop, material storage areas in a warehouse, public areas, etc. may be photographed to obtain the image to be processed.

[0035] At 220, at least one target object is identified and located from the image to be processed using a target detection technology.

[0036] In this embodiment, the target object may be an object that needs to be considered when conducting a 5S management evaluation. In some examples, the trained machine learning model can be used for target detection to identify the target object from the image to be processed and determine its position in the image to be processed. In one example, a training sample containing the target object can be used to supervise the machine learning model so that the trained machine learning model can recognize the above object from a new image. In some examples, the target object may include but is not limited to at least one of the following: personal items such as laptops, mice, keyboards, mobile phones, and water cups, production equipment such as extruders, mixers, granulators, and reactors, storage equipment such as silos, storage tanks, shelves, forklifts, and palletizing robots, and public equipment such as fire extinguishers, exhaust fans, emergency lighting fixtures, cleaning tools, and protective supplies.

[0037] At 230 , local features of each target object are extracted from the image region where each target object is located.

[0038] In this embodiment, the image feature extraction technology can be used to extract the color, shape, texture and other features of the local image area where the target object is located to form the local features of the target object. It can be understood that the local features of the target object can be one (such as color features) or multiple (such as color features, shape features and texture features, etc.). These features are then compared with the features required by the 5S management evaluation rules corresponding to the target object, and the satisfaction is determined based on the comparison results. At 240, the satisfaction between each target object and the corresponding 5S management evaluation rules is determined based on the local features of each target object.

[0039] In this embodiment, different objects may correspond to different 5S management evaluation rules. In some examples, for personal items such as laptops, mice, keyboards, and water cups, the corresponding 5S management evaluation rules may be "remove idle items, place them in fixed positions, and keep them tidy." In some examples, for production equipment such as extruders, mixers, granulators, and reactors, the corresponding 5S management evaluation rules may be "clean up non-essential items around the equipment, ensure that the equipment operating space is unobstructed, the equipment is clearly marked, located in a designated special area, and cleaned and maintained regularly." In some examples, for silos, storage tanks, etc., the corresponding 5S management evaluation rules may be "maintain good sealing, no leakage, clean appearance, clear and accurate labeling, and orderly storage of internal materials"; for shelves, the corresponding 5S management evaluation rules may be "ensure firmness and stability, no deformation or damage, and items are stored in categories and properly labeled"; for forklifts, the corresponding 5S management evaluation rules may be "regular inspection and maintenance, neat appearance, no oil stains and damage, standardized operation, and parking in designated areas"; for palletizing robots, the corresponding 5S management evaluation rules may be "stable operation, no interference from surrounding debris, regular cleaning and maintenance, and identification of its working status and the person responsible for maintenance." In some examples, for fire extinguishers, the corresponding 5S management evaluation rule may be "should be within the validity period, have normal pressure, be in a clear and easily accessible location, and be regularly inspected and maintained"; for exhaust fans, the corresponding 5S management evaluation rule may be "maintain good ventilation, no noise and abnormal vibration, and the surface is clean and free of dust"; for emergency lighting fixtures, the corresponding 5S management evaluation rule may be "should be able to work normally, have sufficient power, be firmly installed, and be tested regularly"; for cleaning tools, the corresponding 5S management evaluation rule may be "placed in the designated area, without damage, clean and free of stains"; for protective equipment, the corresponding 5S management evaluation rule may be "complete and intact, stored in categories, regularly inspected and updated to ensure that they can be used in time when needed".

[0040] In some examples, for the 5S management evaluation rules that instruct to place in a designated position, for example, edge detection technology can be used to extract the edge features of the target object, and the satisfaction with the 5S management evaluation rules can be determined based on whether the target object is located in the designated area. In some examples, the placement areas of production equipment, fire extinguishers, protective equipment, etc. usually have physical markings (such as lines), so that the designated area can be identified from the image. In some examples, for the case where there is no explicit placement area identification, it can be determined whether the target object is located in the designated area by judging whether the target object belongs to the same category as other surrounding target objects. For example, if office items such as laptops, mice, keyboards, and files are placed around a water cup, it can be considered that the water cup is not in the designated area. For another example, if cleaning supplies such as mops, rags, trash cans, and gloves are placed around a broom, it can be considered that the broom is in the designated area.

[0041] In some examples, for the 5S management evaluation rule that indicates keeping clean, for example, the color and texture features of the target object can be extracted using image feature extraction technology, and then classifiers such as support vector machines, neural networks, and decision trees can be used to determine whether the surface of the target object has dust, stains, etc. In some examples, for the 5S management evaluation rule that indicates clear signs, for example, OCR (Optical Character Recognition) technology can be used to perform text recognition on the corresponding signs, and whether the signs are clear can be determined based on the confidence of the text recognition results (such as the confidence of the OCR model output). Furthermore, the text recognition results can also be used to determine the satisfaction with the 5S management evaluation rules that indicate regular maintenance, maintenance, testing, detection and updating, etc., which require recording. In some examples, for the 5S management evaluation rules that indicate ensuring normal operation, for example, image recognition technology can be used to determine whether the indicator light indicating normal operation is on, whether there is a sign such as "under maintenance" hanging on the equipment, so as to determine the satisfaction with the 5S management evaluation rules.

[0042] In this embodiment, the range of satisfaction can be set according to the actual application situation. For example, the satisfaction can be set from 0 to 100 points. The satisfaction between the target object and the corresponding 5S management evaluation rule can be determined in various ways. In some examples, for a certain target object, it is possible to check in sequence whether the corresponding 5S management evaluation rule is met according to the analysis result corresponding to the target object. It can be based on 100 points, and 10 points will be reduced for each non-satisfaction, with a minimum of 0 points. In some examples, different 5S management evaluation rules can also be assigned corresponding scores, and when they are not met, the corresponding scores are deducted. In some examples, the ratio of the number of 5S management evaluation rules met to the total number of management evaluation rules checked can also be multiplied by 100 points to determine the satisfaction of the target object. Thus, the satisfaction corresponding to each target object can be determined.

[0043] At 250 , the obtained satisfactions are aggregated to generate a first score at the object level.

[0044] In this embodiment, various methods can be used to summarize the obtained satisfactions. In some examples, the average value of the satisfactions corresponding to the determined multiple target objects can be determined as the first score of the object level of the image to be processed. For example, the satisfactions corresponding to the laptop computer, mouse, keyboard, and water cup in the image to be processed taken at the office workstation are 80, 90, 80, and 30, respectively, then the first score can be (80+90+80+30) / 4=70. In some examples, different weights can also be assigned to target objects of different categories in advance, so as to determine the first score by weighted summation. For example, in the above example, the weights corresponding to the laptop computer, mouse, keyboard, and water cup can be 30%, 20%, 20%, and 30%, respectively, then the first score can be 80×30%+90×20%+80×20%+30×30%=67.

[0045] At 260 , a 5S management evaluation score of the image to be processed is generated based on the first score.

[0046] In this embodiment, based on the first score, the 5S management evaluation score of the image to be processed can be generated in various ways. In some examples, the first score can be directly determined as the 5S management evaluation score of the image to be processed. In some examples, the 5S management evaluation score of the image to be processed can be generated by using a preset piecewise function according to the interval in which the first score is located.

[0047] It should be understood that all steps and their order in process 200 are exemplary, and the embodiments of the present disclosure will also cover any modification of process 200.

[0048] Reference below Figure 3 , Figure 3 A schematic diagram of an example of a generation process 300 of a 5S management evaluation rule according to an embodiment of the present specification is shown. The generation process 300 is applicable to a case where an image to be processed has a corresponding reference image.

[0049] In this embodiment, the corresponding 5S management evaluation rules may include local feature sets of each reference object extracted from the reference image of the image to be processed. The reference object may be identified and located from the corresponding reference image using target detection technology, and the specific description of the "target object" in step 220 of the above embodiment may be referred to.

[0050] In this embodiment, the image to be processed may have a corresponding reference image. The reference image may be an image that meets the 5S management requirements, for example, an image with a 5S management evaluation score of full marks or close to full marks. In some examples, for areas with relatively fixed uses, such as employee workstations, a product production line, etc., the set reference images may be pre-stored. In some examples, the reference image may be associated with the area to which it corresponds, that is, different areas may correspond to different reference images. For example, the reference image of the office area is different from the reference image of the production area. In some examples, the reference image may have an area identifier. For example, although the workstations of employee A and employee B both belong to the office area, they may have corresponding reference images respectively. For example, reference image X has an area identifier 100, and reference image Y has an area identifier 200. Among them, area identifiers 100 and 200 can be used to indicate the workstations of employee A and employee B, respectively.

[0051] like Figure 3 As shown, the reference image set 310 may include multiple reference images. For one of the reference images 311, the method described in step 220 in the aforementioned embodiment may be first used to identify and locate the reference object 320, so as to ensure the consistency of the standard used to identify and locate the specific object from the image as much as possible. In one example, there may be multiple reference objects 320, such as reference objects 321, 322, etc. Next, the method described in step 230 in the aforementioned embodiment may be used to extract local features of the reference object, so as to ensure the consistency of the standard used to extract local features from the specific object as much as possible. In one example, taking reference object 321 as an example, its local features may include local features 331, 332, 333, etc. All local features of each reference object may constitute a local feature set of the reference object. Similarly, the local features of each reference object may be extracted to form a local feature set of each reference object in the reference image. Thus, the local feature set of each reference object in the reference image may be used to characterize the characteristics that the reference object should have when meeting the 5S management requirements. Furthermore, the obtained local feature set may be used as a 5S management evaluation rule corresponding to the corresponding reference object.

[0052] In some examples, the generation process 300 may be performed after the image to be processed is acquired. After the image to be processed is acquired, the corresponding reference image may be matched according to the target area to which it is targeted, and then the local feature sets of each reference object extracted from the matched reference image may be used. After the target object is identified from the image to be processed, the local feature set matching the target object may be selected from the local feature sets of each reference object extracted as the 5S management evaluation rule corresponding to the target object.

[0053] In some examples, the above generation process 300 can be performed before acquiring the image to be processed. For example, the above operation can be performed in advance for each reference image in the reference image set 310, so that the local feature set corresponding to the reference object in each reference image can be obtained. In these examples, the obtained local feature set as the 5S management evaluation rule can be stored. Therefore, when the image processing method 200 is executed later, it is not necessary to re-execute the process 300 every time to generate the corresponding 5S management evaluation rule. Optionally, the above 5S management evaluation rule can be further stored with the region identifier as an index. For example, the local feature set corresponding to the same reference image can be queried according to the region identifier. In some examples, after the image to be processed is acquired, the corresponding local feature set can be directly matched according to the target area it is targeted at. It can be understood that the matched 5S management evaluation rule can correspond to multiple reference objects in the same reference image. Furthermore, after the target object is identified from the image to be processed, the local feature set matching the target object can be further selected from it as the 5S management evaluation rule corresponding to the target object.

[0054] Through the above method, this scheme creatively proposes to represent the 5S management evaluation rules corresponding to different objects in the form of a local feature set of the reference object in the reference image, and specifically provides a method for generating 5S management evaluation rules when the image to be processed has a corresponding reference image.

[0055] Continue to refer Figure 4 , Figure 4 A flowchart of another example of a generation process 400 of a 5S management evaluation rule according to an embodiment of the present specification is shown. The generation process 400 is applicable to a case where an image to be processed does not have a corresponding reference image.

[0056] At 410 , a set of similar reference images matching the image to be processed is obtained.

[0057] In this embodiment, the same type of reference image set may be composed of reference images containing the target reference object. For the description of the reference image, please refer to the aforementioned Figure 3Related description in the embodiments. The target reference object may belong to the same object category as the target object. In some examples, for areas with less fixed uses, such as temporary storage areas, etc., it is usually impossible to pre-set corresponding reference images. When the target area targeted by the image to be processed belongs to the above situation, a set of similar reference images that match the image to be processed can be obtained. For example, when the target object identified from the image to be processed includes protective gear and a toolbox, reference images containing protective gear and / or toolboxes can be selected from the reference images to form a set of similar reference images that match the image to be processed. For example, a set of similar reference images may include 100 reference images, of which 80 include images of protective gear and 70 include images of toolboxes, that is, 50 images include both images of protective gear and images of toolboxes.

[0058] At 420 , at least one reference object is identified and located from each reference image in a set of similar reference images using a target detection technique.

[0059] At 430 , for each reference image in the same type of reference image set, local features of each reference object are extracted from an image region where each reference object in the reference image is located.

[0060] It should be noted that the above steps 420-430 can refer to Figure 2 In the description of steps 220 - 230 in the embodiment, it is only necessary to replace the image to be processed with each reference image in the same reference image set.

[0061] At 440 , the local features of the reference object are fused in units of reference objects to generate fused local feature sets corresponding to the reference objects as 5S management evaluation rules.

[0062] In this embodiment, various feature fusion methods (such as averaging, etc.) can be used to generate fused local feature sets corresponding to each reference object. In some examples, feature fusion can be performed separately for different types of local features. For example, when the local features include color features, shape features, and texture features, the generated fused local feature set may include fused color features, fused shape features, and fused texture features.

[0063] In some examples, feature fusion can also be performed in a weighted summation manner. For example, a reference image that includes only one reference object (such as protective gear or a tool box) can be assigned a lower weight, while a reference image that includes more reference objects (such as protective gear and a tool box) can be assigned a higher weight. Optionally, the more reference objects included, the closer the overall scene corresponding to the reference image is to the image to be processed, and thus a higher weight should be assigned.

[0064] In some examples, feature fusion can also be performed by clustering first and then averaging. For example, taking the reference object as protective gear, for 80 reference images including images of protective gear in the same reference image set, 80 groups of local features corresponding to the protective gear in each reference image can be obtained. Among them, each group of local features may include but is not limited to at least one of the following: color features, shape features, and texture features. Taking 80 groups of color features as an example, these 80 groups of color features can be clustered, for example, K-means clustering can be used to obtain 3 clusters, each cluster including 50, 20, and 10 groups of color features, respectively. Then, the color features in the cluster with the largest number of color features can be averaged, for example, the above 50 groups of color features are averaged, and the obtained result is determined as the fused color feature. It can be understood that similar methods can also be used for feature fusion of other local features such as shape features and texture features.

[0065] Through the above method, this scheme creatively proposes to use a set of similar reference images selected from reference images to generate local features of reference objects belonging to the same object category as the target object, and further obtains 5S management evaluation rules corresponding to different objects in the form of fused local feature sets by fusion of features based on reference objects, and specifically provides a method for generating 5S management evaluation rules when the image to be processed does not have a corresponding reference image.

[0066] In some examples, the generation process 400 can be performed after the image to be processed is acquired and at least one target object is identified. At this time, the corresponding reference image set of the same type can be matched according to the target area and the identified target object, and then the fused local feature set corresponding to each reference object can be finally generated according to each reference image in the matched reference image set of the same type. In addition, the fused local feature set corresponding to each reference object obtained above can also select the fused local feature set that matches the target object as the 5S management evaluation rule corresponding to the target object.

[0067] In some examples, the above-mentioned generation process 400 can be performed before acquiring the image to be processed. For example, a subset of the reference image set can be used as a similar reference image set in advance, and by performing operations 420-440, a fused local feature set corresponding to each reference object displayed in the similar reference image set can be obtained. In these examples, the obtained fused local feature set as a 5S management evaluation rule can be stored. Therefore, when the image processing method 200 is executed again later, it is not necessary to re-execute process 400 every time to generate the corresponding 5S management evaluation rule. Optionally, the above-mentioned 5S management evaluation rule can be further stored with the object identifier as an index. In some examples, after the image to be processed is acquired and at least one target object is identified, the corresponding fused local feature set can be directly matched according to the identified target object.

[0068] Reference below Figure 5 , Figure 5 A flowchart showing yet another example of an image processing method 500 for 5S management according to an embodiment of the present specification.

[0069] At 510 , an image to be processed captured for a target area is acquired.

[0070] At 520, at least one target object is identified and located from the image to be processed using a target detection technology.

[0071] At 530 , local features of each target object are extracted from the image region where each target object is located.

[0072] At 540 , the satisfaction degree between each target object and the corresponding 5S management evaluation rule is determined according to the local characteristics of each target object.

[0073] At 550 , the obtained satisfaction levels are aggregated to generate a first score at the object level.

[0074] At 560 , it is determined to which region category the target region belongs.

[0075] In this embodiment, the region category to which the target region belongs can be determined in various ways. In some examples, the shooting location information of the image to be processed can be obtained. Then, the region category can be determined based on the preset shooting location information and the region comparison table. In some examples, the shooting location information can be specified by the photographer, for example, the photographer can pre-enter or select "XX workstation", "XX production equipment", "west side of the corridor on the second floor of the office building" from a drop-down list. In some examples, the shooting location information can be the geotagging of the photo generated by the shooting device, that is, the GPS coordinates are embedded in the EXIF ​​data of the photo. In some examples, the target region category can also be determined directly based on the image to be processed with the help of a pre-trained image region recognition model or based on a specific image feature extraction and analysis algorithm. For example, the image to be processed can be provided to a trained convolutional neural network model, which can output the region category to which the target region is most likely to belong. The convolutional neural network model can be trained by a large amount of image data labeled with different region categories, and can learn the visual feature patterns of different regions in the image.

[0076] In some examples, the area category may include at least one of the following: office workstation area, indoor general production area, indoor key production area, outdoor general production area, outdoor key production area, temporary area, experimental area, test area, etc. In some examples, the office workstation area may have specific shapes and layout features such as tables, chairs, and computers, the indoor general production area may have specific production equipment layout and space features, the indoor key production area may have high-precision production equipment and higher environmental requirements, the outdoor general production area may have natural light, large space and specific outdoor facilities, the outdoor key production area may have special protection facilities and equipment layout features, and the temporary area may be relatively messy and the items are not fixed.

[0077] At 570 , a second score of the image level is determined in a corresponding manner according to whether the determined region category belongs to a region category having a corresponding reference image.

[0078] In this embodiment, a reference image library may be pre-stored. In some examples, the reference images in the reference image library may be stored according to regional categories. The target region category determined in step 560 may be compared and matched with the regional category in the reference image library. For example, the category labels in the reference image library are traversed to see whether there is a label record with the same regional category as the target region. If so, it is determined that the determined regional category belongs to the regional category with the corresponding reference image. If not, it is determined that the determined regional category does not belong to the regional category with the corresponding reference image. Afterwards, a corresponding method may be used to determine the second score of the image level. The second score of the image level can be used to reflect the degree of conformity between the image to be processed as a whole and the 5S management evaluation rules.

[0079] In some examples, if the determined region category belongs to a region category having a corresponding reference image, the global similarity between the image to be processed and the corresponding reference image may be determined as the second score.

[0080] In these examples, various global similarity calculation methods can be used to determine the global similarity between the image to be processed and the corresponding reference image. In some examples, a simple similarity calculation method based on a color histogram, based on the difference between the pixel values ​​of the two images, based on a structural similarity index (SSIM), etc. can be used to determine the global similarity. In some examples, a similarity calculation method based on a feature vector can also be used. For example, the feature vectors of the image to be processed and the corresponding reference image can be first extracted respectively, for example, using SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Feature), ORB (Oriented FAST and Rotated BRIEF) algorithm to extract feature points and construct feature vectors, and then calculate the distance between the two feature vectors, such as Euclidean distance or cosine similarity and other indicators to quantify the similarity between the two. It should be noted that the greater the above global similarity, the closer the two images are. This means that if the directly calculated result represents the distance between the two images, the above result needs to be mapped so that it meets the meaning requirements of the second score.

[0081] In some examples, if the determined region category does not belong to a region category having a corresponding reference image, the 5S adjustment score corresponding to the determined region category may be determined as the second score.

[0082] In these examples, due to the differences in 5S management standards and implementation difficulties for different regional categories, the 5S adjustment scores corresponding to different regional categories can be pre-set to reduce the large differences in scores due to regional category factors. For example, for the office workstation area, since it is easier to meet the 5S management standards, it is necessary to be relatively strict in scoring, so a lower 5S adjustment score (for example, 70 points) can be set. For another example, for the production area, since it is more difficult to meet the 5S management standards, it is necessary to be relatively loose in scoring, so a higher 5S adjustment score (for example, 80 points) can be set. In some examples, key production areas often need to apply stricter standards than ordinary production areas, so the 5S adjustment score corresponding to the key production area (for example, 78 points) can be slightly lower than the 5S adjustment score corresponding to the ordinary production area (for example, 82 points). For another example, due to its temporary nature and uncertainty, the corresponding 5S adjustment score of the temporary area can be set based on the historical score (for example, average value) of the temporary area.

[0083] At 580 , a 5S management evaluation score of the image to be processed is generated based on the first score and the second score.

[0084] In this embodiment, the 5S management evaluation score of the image to be processed can be generated by combining the first score and the second score in various ways. In some examples, the 5S management evaluation score of the image to be processed can be generated by weighted summation. Thus, the final 5S management evaluation score can more comprehensively and objectively reflect the comprehensive performance level of the scene represented by the image to be processed in terms of 5S management.

[0085] It should be noted that the above steps 510-550 can refer to the corresponding description of steps 210-250 in the above embodiment, which will not be repeated here. It can be understood that steps 560-570 can also be performed before any step in steps 520-550.

[0086] Continue to refer Figure 6 , Figure 6 FIG. 6 is a flowchart showing an example of a process 600 for determining a global similarity between an image to be processed and a corresponding reference image according to an embodiment of the present specification. The process 600 may be Figure 5 An exemplary implementation of step 570 in FIG.

[0087] At 610 , a first key point and a second key point are determined from the image to be processed and the corresponding reference image, respectively.

[0088] In this embodiment, algorithms such as SIFT, SURF, and ORB can be used to determine key points. For example, taking the SIFT algorithm as an example, extreme point detection can be first performed in different scale spaces of the image to be processed and the corresponding reference image, and by constructing a Gaussian difference pyramid, local extreme points can be searched at different levels of the pyramid as candidate key points. Then, low-contrast and unstable edge response points can be removed from these candidate key points to obtain a relatively stable and representative first key point (in the image to be processed) and a second key point (in the reference image). These key points determined in the above manner have good properties such as scale invariance and rotation invariance, and can effectively characterize the local feature information of the image under different viewing angles and scale changes. For example, in the reference image corresponding to the production area, the corners of the equipment, the position of specific logos, etc. may be detected as key points.

[0089] At 620 , based on the matching point pairs formed by the first key point and the second key point, a perspective transformation matrix for transforming the image to be processed into the corresponding reference image is determined.

[0090] In this embodiment, after obtaining the first key point and the second key point, these key points can be matched by feature descriptors to form matching point pairs. For example, SIFT feature descriptors can be used to describe the local feature information around each key point, and then the similarity between different key point descriptors (such as Euclidean distance and other metrics) is calculated, and the key point pairs whose similarity meets a certain threshold are determined as matching point pairs. Then, these matching point pairs can be used to calculate the perspective transformation matrix using the random sampling consensus (RANSAC) algorithm. The RANSAC algorithm can estimate the transformation model by randomly selecting a subset of the matching point pairs for multiple times, and then the number of inliers of the transformation model (i.e., the number of matching point pairs that conform to the transformation model) can be calculated. By repeating the above process, the perspective transformation matrix corresponding to the transformation model with the largest number of inliers is finally selected as the perspective transformation matrix for transforming the image to be processed to the reference image. Thereby, the wrongly matched point pairs can be effectively removed, and the accuracy and reliability of the perspective transformation matrix can be improved. The above perspective transformation matrix is ​​usually a 3x3 homography matrix (Homography), which describes the projection relationship between images. In one example, the above process can be efficiently implemented using relevant functions (such as findHomography function) in an image processing library (such as OpenCV, etc.).

[0091] At 630 , the image to be processed is perspective transformed using a perspective transformation matrix to obtain a perspective transformed image that is aligned with the corresponding reference image.

[0092] In this embodiment, the perspective transformation matrix determined in step 620 can be used to perform perspective transformation on the image to be processed so as to align it with the corresponding reference image. In one example, for any pixel coordinate (x, y) in the image to be processed, its homogeneous coordinate can be expressed as [x, y, 1] T . Let the perspective transformation matrix be H. The homogeneous coordinates of the pixel point (x, y, 1) can be transformed by matrix multiplication. T Multiply it with the perspective transformation matrix H to get the homogeneous coordinates [x', y', 1] after perspective transformation T , and then the pixel coordinates (x', y') after perspective transformation can be obtained. By performing the above perspective transformation operation on each pixel in the image to be processed, a perspective transformed image aligned with the corresponding reference image in perspective effect can be obtained. In an example, the above process can be efficiently implemented using relevant functions (such as warpPerspective function) in an image processing library (such as OpenCV, etc.).

[0093] At 640, a global similarity between the perspective transformed image and a corresponding reference image is determined.

[0094] In this embodiment, the global similarity can be determined by referring to the relevant description of the aforementioned step 572 and adopting a simple similarity calculation method based on a color histogram, based on the difference between the pixel values ​​of two images, based on a structural similarity index, etc.

[0095] In the above manner, this solution organically combines perspective transformation with the determination of global similarity, so that the image to be processed can be aligned with the corresponding reference image as much as possible in terms of geometric shape, position, etc., and by transforming the objects in the image from different perspective angles to a unified perspective, a more accurate comparison and evaluation of the similarity between images and the 5S management status can be achieved. This is especially suitable for processing images with depth information or large differences in shooting angles that are common in manufacturing enterprises. For example, when analyzing images of factory workshops or warehouses with different shooting angles for 5S management, it can be determined whether the shelves are neatly arranged in different images, whether the passages are unobstructed, etc. For another example, when analyzing images of office areas for 5S management, objects such as tables, chairs, and filing cabinets in the two images can be made to correspond in spatial position, thereby effectively reducing the scoring error caused by different perspectives.

[0096] Continue to refer Figure 7 , Figure 7 FIG. 7 is a flowchart showing an example of a process 700 for generating a 5S management evaluation score according to an embodiment of the present specification. The process 700 may be Figure 5 An exemplary implementation of step 580 in FIG.

[0097] At 710 , a determination is made as to whether the first score and the second score match.

[0098] In this embodiment, if the first score and the second score are not much different, for example, not exceeding a preset threshold (for example, 10 points), it can be determined that the first score and the second score match. If the difference exceeds the above preset threshold, it can be determined that the first score and the second score do not match.

[0099] If 710 determines that there is no match, the following steps 720-740 are executed.

[0100] At 720 , the perspective transformed image and the corresponding reference image are provided to a non-scoring sensitive region identification model to determine whether there is a non-scoring sensitive region.

[0101] In this embodiment, the non-scoring sensitive area is used to indicate an area that has no effect on whether the 5S management evaluation rules are met despite the difference between the perspective transformed image and the corresponding reference image. In some examples, the non-scoring sensitive area recognition model may be a model trained based on a machine learning method. The perspective transformed image and the corresponding reference image may be provided to the non-scoring sensitive area recognition model. If there is a non-scoring sensitive area, the non-scoring sensitive area recognition model may mark the corresponding non-scoring sensitive area in the perspective transformed image and the corresponding reference image. If there is no non-scoring sensitive area, there is no corresponding non-scoring sensitive area mark in the perspective transformed image and the corresponding reference image. In some examples, the non-scoring sensitive area recognition model may be obtained by training an image as a training sample. The image pair as a training sample may be composed of a first sample image and a second sample image. The content presented in the first sample image and the second sample image may be different, but the scores corresponding to the first sample image and the second sample image that have been manually annotated are substantially the same. For example, for the corridor of an office building, if the decorative painting on the wall changes from painting A to painting B, the area where the decorative painting is located may be identified as a non-scoring sensitive area. For another example, for the same workstation, if the content displayed on the laptop screen should not affect the 5S management evaluation score, the laptop screen area can be identified as a non-scoring sensitive area. For another example, for the production area, some areas on the ground with slight marks caused by normal production wear and tear can also be identified as non-scoring sensitive areas if they do not affect the 5S key indicators such as the cleanliness, safety of the workshop, and the orderly placement of equipment and materials.

[0102] If 720 determines that it exists, execute steps 730-740.

[0103] At 730, at least one of the first score and the second score is modified based on the identified non-score sensitive area.

[0104] In this embodiment, at least one of the first score and the second score can be corrected according to the position of the non-scoring sensitive area in the image, the area ratio, the corresponding satisfaction of the determined target object, the relative size of the first score and the second score, etc. In some examples, if the position of the non-scoring sensitive area in the image is the central area or the key area of ​​the 5S management evaluation or the area ratio is large (for example, exceeding the area ratio threshold, such as 10%), it means that it will affect the first score and the second score at the same time, and the second score representing the global is more affected, then the first score and the second score can be corrected with the first upward adjustment strength, and the upward adjustment strength of the second score is greater than the upward adjustment strength of the first score. In some examples, if the position of the non-scoring sensitive area in the image is the edge area or the non-key area of ​​the 5S management evaluation or the area ratio is small, the first score and the second score can be corrected with the second upward adjustment strength, and the second upward adjustment strength is less than the first upward adjustment strength. In some examples, the correction of the first score can also be based on the recalculation of the first score of the image to be processed on the basis of removing the satisfaction corresponding to the target object corresponding to the non-scoring sensitive area. In some examples, the smaller score between the first score and the second score may be corrected to be an average of the first score and the second score.

[0105] At 740 , a 5S management evaluation score of the image to be processed is generated according to the correction result.

[0106] In this embodiment, the 5S management evaluation score of the image to be processed may be generated with reference to the description of step 270 , and only the first score and / or the second score need to be replaced with the corresponding correction result.

[0107] In the above manner, the present solution creatively combines the deviation between the first score and the second score and the identification of non-scoring sensitive areas, thereby reducing the deviation caused by factors unrelated to 5S management, thereby achieving more accurate image comparison and generating more objective 5S management evaluation scores.

[0108] use Figure 1-Figure 7 The image processing method for 5S management disclosed in the can obtain the first score of the object level by the satisfaction between the extracted local features and the corresponding 5S management evaluation rules, so as to provide a reference for the final 5S management evaluation score. Compared with the conventional method of directly using the standard image of the area for comparison, this solution analyzes the compliance between the image to be processed and the corresponding 5S management evaluation rules from the perspective of image understanding, so that it can be suitable for scenes without corresponding standard images, and can also greatly reduce the storage volume of standard images.

[0109] Figure 8 A block diagram showing one example of an image processing apparatus 800 for 5S management according to an embodiment.

[0110] The image processing device 800 may include: an image acquisition unit 810, configured to acquire an image to be processed taken for a target area; a target detection unit 820, configured to identify and locate at least one target object from the image to be processed using target detection technology; a feature extraction unit 830, configured to extract local features of each target object from the image area where each target object is located; a first score determination unit 840, configured to determine the degree of satisfaction between each target object and the corresponding 5S management evaluation rule based on the local features of each target object; summarize the obtained degrees of satisfaction to generate a first score at the object level; and a final score generation unit 850, configured to generate a 5S management evaluation score of the image to be processed based on the first score.

[0111] In addition, the image processing apparatus 800 may further include any other modules configured to perform any operation of the image processing method for 5S management according to the above-mentioned embodiment of the present disclosure.

[0112] Reference above Figures 1 to 8 , an embodiment of an image processing method and apparatus for 5S management according to an embodiment of this specification is described.

[0113] The image processing device for 5S management in the embodiment of this specification can be implemented by hardware, software, or a combination of hardware and software. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the device in which it is located reading the corresponding computer program instructions in the memory into the memory and running it. In the embodiment of this specification, the image processing device for 5S management can be implemented, for example, using an electronic device.

[0114] Fig. 9 A schematic diagram showing an example of an image processing apparatus 900 for 5S management according to an embodiment of the present specification.

[0115] like Fig. 9 As shown, the image processing apparatus 900 for 5S management may include at least one processor 910, a memory (e.g., a non-volatile memory) 920, a memory 930, and a communication interface 940, and the at least one processor 910, the memory 920, the memory 930, and the communication interface 940 are connected together via a bus 950. The at least one processor 910 executes at least one computer-readable instruction (i.e., the above-mentioned element implemented in the form of software) stored or encoded in the memory.

[0116] In one embodiment, computer executable instructions are stored in a memory, which, when executed, enable at least one processor 910 to: obtain an image to be processed that is taken of a target area; identify and locate at least one target object from the image to be processed using target detection technology; extract local features of each target object from the image area where each target object is located; determine the degree of satisfaction between each target object and the corresponding 5S management evaluation rule based on the local features of each target object; summarize the obtained degrees of satisfaction to generate a first score at the object level; and generate a 5S management evaluation score for the image to be processed based on the first score.

[0117] It should be understood that the computer executable instructions stored in the memory, when executed, cause at least one processor 910 to perform the above combined operations in various embodiments of the present specification. Figure 1-Figure 7 Describes the various operations and functions.

[0118] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0119] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above-mentioned embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute a part of the present invention.

[0120] The computer program code required for the operation of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages, such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB, NET and Python, conventional procedural programming languages ​​such as C language, Visual Basic 2003, Perl, COBOL 2002, PHP and ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and another part on the remote computer, or all on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service, such as software as a service (SaaS).

[0121] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0122] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0123] Not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined according to needs. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0124] The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described techniques. However, the techniques may be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0125] The optional implementation modes of the embodiments of the present specification are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present specification are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present specification, various simple modifications can be made to the technical solutions of the embodiments of the present specification, and these simple modifications all belong to the protection scope of the embodiments of the present specification.

[0126] The above description of the contents of this specification is provided to enable any person of ordinary skill in the art to implement or use the contents of this specification. Various modifications to the contents of this specification will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the contents of this specification. Therefore, the contents of this specification are not limited to the examples and designs described herein, but are consistent with the widest range of principles and novel features disclosed herein.

Claims

1. An image processing method for 5S management, comprising: Acquire an image to be processed that is taken for a target area; Using target detection technology to identify and locate at least one target object from the image to be processed; Extracting local features of each target object from the image region where each target object is located; According to the local characteristics of each target object, determine the satisfaction degree between each target object and the corresponding 5S management evaluation rules; Aggregate the obtained satisfactions to generate a first score at the object level; as well as A 5S management evaluation score of the image to be processed is generated based on the first score.

2. The image processing method according to claim 1, wherein: If the image to be processed has a corresponding reference image, the corresponding 5S management evaluation rule includes a local feature set of each reference object extracted from the reference image of the image to be processed, wherein the reference object is identified and located from the corresponding reference image using target detection technology.

3. The image processing method according to claim 1, wherein: If the image to be processed does not have a corresponding reference image, The 5S management evaluation rules are generated through the following steps: Acquire a set of similar reference images that match the image to be processed, wherein the set of similar reference images consists of reference images containing a target reference object, and the target reference object belongs to the same object category as the target object; Using target detection technology to identify and locate at least one reference object from each reference image in the set of similar reference images; For each reference image in the same type of reference image set, extracting local features of each reference object from an image region where each reference object in the reference image is located; and Taking the reference object as a unit, the local features of the reference object are fused to generate a fused local feature set corresponding to each reference object as a 5S management evaluation rule.

4. The image processing method according to claim 1, wherein: The image processing method further comprises: Determining the area category to which the target area belongs; Determine a second score of the image level in a corresponding manner according to whether the determined region category belongs to a region category having a corresponding reference image, Generating the 5S management evaluation score of the image to be processed based on the first score includes: A 5S management evaluation score of the image to be processed is generated based on the first score and the second score.

5. The image processing method according to claim 4, wherein: The determining the second score of the image level in a corresponding manner according to whether the determined region category belongs to a region category having a corresponding reference image comprises: If the determined region category belongs to a region category having a corresponding reference image, the global similarity between the image to be processed and the corresponding reference image is determined as the second score.

6. The image processing method according to claim 5, wherein: The global similarity between the image to be processed and the corresponding reference image is determined by the following steps: Determining a first key point and a second key point from the image to be processed and a corresponding reference image respectively; Determining a perspective transformation matrix for transforming the image to be processed into a corresponding reference image based on a matching point pair formed by the first key point and the second key point; Performing perspective transformation on the image to be processed using the perspective transformation matrix to obtain a perspective transformed image aligned with a corresponding reference image; as well as A global similarity between the perspective transformed image and a corresponding reference image is determined.

7. The image processing method according to claim 6, wherein: The generating the 5S management evaluation score of the image to be processed based on the first score and the second score comprises: determining whether the first score and the second score match; In response to a mismatch, Providing the perspective transformed image and the corresponding reference image to a non-scoring sensitive area recognition model to determine whether there is a non-scoring sensitive area, wherein the non-scoring sensitive area is used to indicate an area that has no effect on whether the 5S management evaluation rules are met although there is a difference between the perspective transformed image and the corresponding reference image; If so, at least one of the first score and the second score is corrected according to the identified non-scoring sensitive area; and a 5S management evaluation score of the image to be processed is generated according to the correction result.

8. The image processing method according to claim 4, wherein: The determining the second score of the image level in a corresponding manner according to whether the determined region category belongs to a region category having a corresponding reference image comprises: If the determined area category does not belong to the area category having a corresponding reference image, the 5S adjustment score corresponding to the determined area category is determined as the second score.

9. An image processing device for 5S management, comprising: An image acquisition unit, configured to acquire an image to be processed taken for a target area; A target detection unit, configured to identify and locate at least one target object from the image to be processed using a target detection technology; A feature extraction unit is configured to extract local features of each target object from the image area where each target object is located; The first score determination unit is configured to determine the satisfaction between each target object and the corresponding 5S management evaluation rule according to the local characteristics of each target object; summarize the obtained satisfactions to generate a first score at the object level; as well as The final score generating unit is configured to generate a 5S management evaluation score of the image to be processed based on the first score.

10. An image processing device for 5S management, comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the image processing method for 5S management as described in any one of claims 1 to 8.

Images