System and method for malfunction prediction from exhaust smoke images in construction machinery

Abstract

The invention relates to a method and a system operating according to the method, based on person-independent algorithmic foundations which involves analyzing and processing exhaust smoke images in construction machinery to predict malfunctions through machine learning methods and sharing the results with relevant business units; predicting malfunctions over long periods through a mobile application operated on a mobile device and a platform running on an information processing device.

Description

[0001] SYSTEM AND METHOD FOR MALFUNCTION PREDICTION FROM EXHAUST SMOKE IMAGES IN CONSTRUCTION MACHINERY

[0002] Technical Field

[0003] The invention relates to a method and a system operating according to the method, based on person-independent algorithmic foundations which involves analyzing and processing exhaust smoke images in construction machinery to predict malfunctions through machine learning methods and sharing the results with relevant business units; predicting malfunctions over long periods through a mobile application operated on a mobile device and a platform running on an information processing device.

[0004] State Of The Art

[0005] Exhaust gas emission is a condition caused by chemical gases released into the air as a result of the combustion of gasoline and diesel fuel used in vehicles. The disposal of these chemical compounds, which are formed as a result of the combustion of the fuel used in motor vehicles in the cylinders, differs in diesel and gasoline vehicles. Hydrocarbons, carbon monoxide, and nitrogen oxides are common wastes produced by both diesel and gasoline vehicles, while diesel vehicles additionally produce particles. Carbon monoxide, carbon dioxide, nitrogen oxide, other hydrocarbon gases and heavy metals such as lead are released into the air in gaseous form through exhaust. Exhaust gases cause irreparable damage to the environment.

[0006] Exhaust emissions are regularly monitored, but malfunctions that occur outside the controls affect exhaust emissions. The exhaust smoke emitted by a motor vehicle can provide clues about malfunctions to the user. The color of smoke from the exhaust smoke of motor vehicles can be indicative of certain malfunctions. In such cases, recognizing the sign in question and intervening early can prevent serious problems and costs that may occur in the future and significantly reduce air pollution.

[0007] In the current state of the art, exhaust smoke color does not provide a clear detection for malfunction detection. Some exhaust smoke colors have been associated with malfunctions, but this is not the first and definitive method of malfunction detection. Failure to detect malfunctions before they occur by the user can lead to larger malfunctions, operational disruptions, the release of harmful gases into the air, and an increase in air pollution.

[0008] Patent document CN 111080621 A addresses the detection of ground damage in railway wagons using malfunction images. In the dynamic vehicle inspection processes, it is aimed to increase efficiency, reduce labor cost and finally to detect a malfunction without depending on the quality and fatigue of the personnel. Image processing and deep learning were used in this malfunction detection process. However, the solution described therein does not solve the problem of mobile detection of the malfunction by the user personnel, without the need for the knowledge and experience of the technical personnel to detect the malfunction and prevent the loss of time to come to the service.

[0009] Patent document CN108805868A describes an image processing method and a malfunction detection method for detecting malfunctions in equipment parts of an electric service vehicle, which carries a type of electric vehicle-mounted undercarriage imaging processing method. It also discloses an image processing method for equipment malfunction detection and a malfunction detection method.

[0010] Patent document CN111080668A relates to a method and system for the detection of wear and damage of brake pads in railway vehicles. It detects wear by comparing the image of the brake pad before it is installed and the image during operation.

[0011] The above-mentioned patent documents and many more disclose failure prediction systems and methods. However, these disclosed inventions do not provide a comparison for exhaust smoke or offer a solution such as malfunction detection.

[0012] Overall, there are efforts in the current state of the art to detect malfunctions that have been developed or are in use. However, no solution has yet emerged that enables the user to diagnose the malfunction on a mobile basis without going to the service centre and to diagnose the malfunction from the exhaust smoke colour. In order to overcome the disadvantages of the known state of the art, new methods need to be developed.

[0013] Objectives and Brief Description of the Invention

[0014] The purpose of the invention is to obtain a method and a system that operates according to the method, enabling rapid malfunction detection and prediction based solely on exhaust smoke color without the need for additional equipment in construction machinery.

[0015] Another goal of the invention is to optimize manual processes through a mobile application, grounded on person-independent algorithmic foundations, thus ensuring versatile and digital monitoring.

[0016] Another aim of the invention is to minimize the downtime when machines are inoperable and to prevent customers from wasting time by coming to the service centre.

[0017] Another aim of the invention is to protect the environment by preventing harmful gases emitted from the exhaust smoke.

[0018] With the malfunction detection, logistics and personnel costs are reduced by determining the need for technical personnel and spare parts according to the nature of the malfunction. By sending the right personnel for the right job, planning errors are minimized, and the time spent waiting for tasks is improved. Therefore, the invention also provides advantages in terms of labour, time, and cost.

[0019] To achieve the above-mentioned objectives, the invention is a system that allows for detecting malfunction by analysing exhaust smoke images in construction machinery and it comprises:

[0020] - at least one mobile device containing a mobile application that identifies smoke images that could cause malfunction and sends the smoke image to a database,

[0021] - at least one CRM system containing an internet platform for sharing malfunction information with relevant business units and maintaining various data,

[0022] - a data warehouse enabling data exchange between different platforms,

[0023] - at least one server facilitating the development of prediction algorithms using machine learning methods,

[0024] - learning component and a prediction component allowing for continuous improvement processes for the mentioned server.

[0025] The system also comprises at least one testing platform and prediction component, enabling the evaluation of prediction algorithm outcomes by business units based on predefined criteria and a CRM system that is instantaneously integrated with the prediction component. The invention is also a method that enables malfunction detection by analysing the images of exhaust smoke in construction machinery and includes the following process steps:

[0026] - recording the smoke image from the exhaust of the suspected malfunctioning machine via the application on a mobile device and sending it to the data warehouse,

[0027] - sending the image data stored in the database to server via a data warehouse,

[0028] - analysing the data received from the data warehouse on the server using image processing algorithms to determine which component is malfunctioning.

[0029] The method also comprises the step of sending the malfunction detection information recorded in the CRM system to the mobile device of the relevant person via the data warehouse.

[0030] The method of the invention further comprises the steps of real time interaction between the mentioned server and the image database via the data warehouse, allowing for continuous improvement of malfunction prediction results using image processing algorithms.

[0031] In the alternative structure of the method according to the invention, it involves the process steps of transmitting the data processed within the server's learning component to the prediction component at specific intervals, with the prediction component being seamlessly integrated with the CRM system in real-time.

[0032] Brief Description of the Figures

[0033] In Figure-1 , the system components of the invention and their relationship are given.

[0034] Figure-2 provides a flow diagram showing the process steps related to the method of the invention.

[0035] Reference Numbers

[0036] 100. System

[0037] 10. Mobile device

[0038] 11. Mobile application 20. Datebase

[0039] 21. Data Warehause

[0040] 30. System server

[0041] 31. Learning component

[0042] 32. Prediction component

[0043] 40. CRM system

[0044] 200. Method

[0045] 201. Sending the exhaust smoke image of the machine that is thought to be malfunctioning to the database by taking the image through the application on the mobile device.

[0046] 202. Sending the image data stored in the database to the server via the data warehouse,

[0047] 203. Analysing the colour, density and smoke output shape of the data coming from the database on the server with image processing algorithms to determine which component the malfunction is in.

[0048] 204. After determining in which component is malfunctioned, sending the malfunction detection information from the server to the CRM system via the data warehouse and recording it.

[0049] 205. Sending the malfunction detection information recorded in the CRM system to the relevant person's mobile device and data processing device via the data warehouse.

[0050] Detailed Description of the Invention

[0051] The invention discloses a malfunction prediction system (100) developed to determine the presence of a malfunction and, if there is a malfunction, its origin from which part, by analysing the colour of the exhaust smoke emitted by construction machinery during operation and discloses the operational method (200) of this system (100).

[0052] The exhaust smoke of construction machines is recorded by means of a platform running on a mobile application (11) on the mobile devices (10) of personnel such as machine operators, technicians, or customers. The exhaust smoke image and image data from the data warehouse (21) are analysed by image processing algorithms on the server (30). If a malfunction is detected because of the analysis, the malfunction information (presence and source of the malfunction) is notified to the mobile devices (10) of the personnel or customers through the mobile application (11).

[0053] The system (100) components and their relationships for the invention are given in Figure 1. The system (100) comprises

[0054] - a mobile device (10) containing a mobile application (11) that images and identifies exhaust smoke and sends the images to a data warehouse (21),

[0055] - a database (20) that stores images from the mobile application (11),

[0056] - a data warehouse (21) that allows the exchange of data between different platforms with the database (20),

[0057] - a server (30) that analyses the images of smoke colours received through the data warehouse (21),

[0058] - a prediction component (32) in said server (30), which runs the algorithm that determines the presence and source of the malfunction,

[0059] - a learning component (31) within the mentioned server (30) that continuously applies improvement processes to identify the most suitable algorithm for the mentioned prediction component (32)

[0060] - a CRM system (40) in which the results from the prediction component (32) are recorded through the data warehouse (21), as well as previous malfunction and maintenance records.

[0061] The learning component (31) mentioned above also functions as a test platform, allowing the results of the prediction algorithms to be measured by business units according to predefined criteria.

[0062] The operational method (200) of the invention for predicting malfunctions from the exhaust smoke image in construction machinery comprises the following steps

[0063] - capturing the exhaust smoke image of the machine suspected of malfunctioning via the application on the mobile device (10) and sending (201) it to the data warehouse (20),

[0064] - transmitting (202) the image recorded in the database (20) to the server (30) through a data warehouse (21), - analysing the data from the database (20) on the server (30) using image processing algorithms to determine the presence of a malfunction and which component it is in by analysing colour, density, and smoke output shape (203),

[0065] - after determining the presence of a malfunction and which component it is in, sending the malfunction detection information from the server (30) to the CRM system (40) via the data warehouse (21) for recording (204),

[0066] - sending the malfunction detection information recorded in the CRM system (40) to the relevant person's mobile device (10) and / or any computing device containing the mentioned mobile application (11) via the data warehouse (21) (205).

[0067] While data is processed daily within the learning component (31) of the mentioned server (30), continuous improvement efforts are conducted in the malfunction prediction system (100) by means of data flow to the learning component (31). Here, the exhaust smoke image data obtained from the image database (20) undergoes colour analysis via image processing algorithms determined by the learning component (31), enabling analysis through the prediction component (32). Artificial neural network algorithms are utilized in the analysis processes occurring on the server (30).

[0068] During the analysis of smoke images, variable comparison techniques are also employed using data on smoke colours. Exhaust smoke images are categorized into four main headings based on grey, white, black, and blue colours. Processing data such as machine type, model, operating hours, working conditions, maintenance history, etc., through machine learning methods enables maintenance prediction and contributes to the development of data analytics infrastructure related to construction machinery.

[0069] Within the server (30), the learning component (31) undergoes a process of learning from its own errors, while the prediction component (32) ensures real-time processing of data and transfers it to the data warehouse (21) and the CRM system (40). After determining the presence of a malfunction and its location, the malfunction detection information received from the server (30) via the data warehouse (21) is sent to the CRM system (40) for recording. The malfunction detection information recorded in the CRM system (40) is then sent to the customer's or the technical personnel's mobile device (10) through the data warehouse (21). This allows users to quickly and conveniently learn which component the malfunction is in. In a preferred configuration of the system, any computing device can be used instead of a mobile device (10). It is sufficient for this computing device to contain the mobile application (11).

Claims

CLAIMS1. A system (100) that enables malfunction detection by analysing exhaust smoke images in construction machinery, characterized by comprising- at least one mobile device (10) comprising a mobile application (11) providing identification of smoke images that may be caused from malfunction and sending the smoke image to the database (20),- at least one CRM system (40) comprising an internet platform for sharing malfunction information with relevant departments and keeping various data,- a data warehouse (21) enabling data exchange between different platforms,- at least one server (30) enabling the development of prediction algorithms using machine learning methods,- a learning component (31) and a prediction component (32) allowing for continuous improvement of the server (30).

2. A system (100) in accordance with Claim 1 , wherein it comprises a CRM system (40) featuring real-time integration with the prediction component (32).

3. A method enabling malfunction detection by analysing exhaust smoke images of construction machinery, characterised by comprising the steps of- recording the exhaust smoke image of the suspected malfunction machine through application on a mobile device (10) and sending it to a database (20) (201),- sending the recorded image data stored in the database (20) to a server (30) via a data warehouse (21) (202),- analysing the data received from the data warehouse (21) on the server (30) using image processing algorithms and determining malfunction component (203).

4. The method according to claim 3, wherein it further comprises the steps of- after determining the malfunction component, sending the malfunction detection data via the data warehouse (21) to a CRM system (40) for recording (204),- transmitting the recorded malfunction detection information from the CRM system (40) to the mobile device (10) through the data warehouse (21) (205).

5. The method according to claim 3, wherein the analysis (203) comprises the steps of- real-time interaction of the server (30) with the image database (20) through the data warehouse (21),- thus, enabling continuous improvement of malfunction prediction results through the utilization of image processing algorithms on the image data.

6. The method according to claim 5, wherein the analysis (203) further comprises the steps of- the transfer of data processed within a learning component (31) in the server (30) to a prediction component (32) at specific intervals,- the prediction component (32) being integrated with the CRM system (40) in real-time.

Images