Image processing method, device, storage medium and electronic device
By dividing image processing tasks into linear communication tasks in the cloud environment and calling microservices, efficient automated scheduling of multi-visual tasks is achieved, solving the problems of low image processing efficiency and relying on manual intervention in the existing technology, and improving the cloud environment image processing efficiency.
Patent Information
- Application Number
- CN202210229536.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-03-10
Smart Images

Figure CN116775232B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular to image processing methods, devices, storage media, and electronic devices. Background Art
[0002] With the development of the Internet, the number of people using the Internet and the images spread on the Internet have increased rapidly. Correspondingly, the visual tasks for images that rely on Internet transmission are also diverse, such as identifying special positions of images, classifying images to determine whether the images involve certain sensitive content, etc. The relevant technologies lack a comprehensive plan for these visual tasks, and also lack a management and comprehensive scheduling plan for these visual tasks, which makes the implementation of these visual tasks rely on a large degree of human intervention, resulting in low efficiency in visual task execution and difficulty in meeting the visual analysis needs under large amounts of data in cloud environments. Summary of the invention
[0003] In order to solve at least one of the above technical problems, the embodiments of the present application provide an image processing method, device, storage medium and electronic device.
[0004] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:
[0005] Acquire an image and configuration data for the image, the configuration data being used to define at least one first visual task for processing the image, wherein any first visual task is a linear communication task including at least one subtask, each subtask pointing to a corresponding microservice, and the microservice running based on a corresponding visual processing model;
[0006] According to the linear communication model corresponding to the second visual task, the first subtask is determined, and the microservice corresponding to the first subtask is called so that the microservice corresponding to the first subtask processes the image to obtain a first processing result corresponding to the first subtask, and the second visual task is any first visual task among the at least one first visual task.
[0007] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:
[0008] a data acquisition module, configured to acquire an image and configuration data for the image, wherein the configuration data is used to define at least one first visual task for processing the image, wherein any first visual task is a linear communication task including at least one subtask, each subtask pointing to a corresponding microservice, and the microservice runs based on a corresponding visual processing model;
[0009] A data processing module is used to determine a first subtask according to a linear communication model corresponding to a second visual task, and call a microservice corresponding to the first subtask so that the microservice corresponding to the first subtask processes the image to obtain a first processing result corresponding to the first subtask, wherein the second visual task is any first visual task among the at least one first visual task.
[0010] On the other hand, an embodiment of the present application provides an image processing system, the system comprising:
[0011] A business layer component, used for acquiring the image;
[0012] an access layer component, configured to set configuration data for the image, the configuration data being used to define at least one first visual task for processing the image, and to configure a linear communication model corresponding to each first visual task, wherein any first visual task is a linear communication task including at least one subtask, each subtask pointing to a corresponding microservice, and the microservice running based on the corresponding visual processing model;
[0013] Microservice cluster, used to manage each microservice;
[0014] Model layer components, used to manage various visual processing models;
[0015] The access layer component is also used to trigger the running of the second visual task, so that the second visual task determines the first subtask according to the corresponding linear communication model, calls the microservice corresponding to the first subtask, so that the microservice corresponding to the first subtask processes the image to obtain the first processing result corresponding to the first subtask, and the second visual task is any first visual task among the at least one first visual task.
[0016] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned image processing method.
[0017] On the other hand, an embodiment of the present application provides an electronic device, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-mentioned image processing method by executing the instructions stored in the memory.
[0018] On the other hand, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implement the above-mentioned image processing method when executed by a processor.
[0019] The image processing method provided by the present application can provide a framework for collaboratively scheduling multiple visual tasks. Each visual task running within the framework is run based on a linear communication model, and the subtasks in each visual task are encapsulated in the form of microservices. The same subtasks between different visual tasks can be reused, so that the services associated with each visual task can be reasonably utilized to ensure the collaborative and efficient operation of multiple visual tasks. The system scheduling of multiple visual tasks under this framework no longer relies on human intervention, and can be implemented fully automatically and efficiently, significantly improving the image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 It is a schematic diagram of a feasible implementation framework of the image processing method provided in the embodiments of this specification;
[0022] Figure 2 is another feasible implementation framework schematic diagram of the image processing method provided in the embodiments of this specification;
[0023] Figure 3 The embodiment of the present application provides Figure 2 The data processing flow chart of the framework;
[0024] Figure 4 It is a flowchart of an image processing method provided in an embodiment of the present application;
[0025] Figure 5 It is a schematic diagram of a scheduling method of a first visual task provided in an embodiment of the present application;
[0026] Figure 6 Another scheduling method of a first visual task provided by an embodiment of the present application is a schematic diagram;
[0027] Figure 7 This is a schematic diagram of the model distribution corresponding to the microservices provided in the embodiment of the present application;
[0028] Figure 8 is a schematic diagram of image preprocessing provided in an embodiment of the present application;
[0029] Fig. 9 It is a schematic diagram of the selection of preprocessing operations in the preprocessing link provided in the embodiment of the present application;
[0030] Fig.10 It is a schematic diagram of the degradation operation in the preprocessing step provided in the embodiment of the present application;
[0031] Fig.11 is a schematic diagram of a face detection task framework provided in an embodiment of the present application;
[0032] Fig.12 This is a schematic diagram of the face detection task flow provided by the embodiment of the present application;
[0033] Fig.13 It is a schematic diagram of resource scheduling provided in an embodiment of the present application;
[0034] Fig.14 is a block diagram of an image processing device provided in an embodiment of the present application;
[0035] Fig.15 It is a schematic diagram of the hardware structure of a device provided in an embodiment of the present application for implementing the method provided in an embodiment of the present application;
[0036] Fig.16 It is a schematic diagram of the hardware structure of another device provided in an embodiment of the present application for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.
[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0039] In order to make the purpose, technical solution and advantages disclosed in the embodiments of the present application more clearly understood, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and are not used to limit the embodiments of the present application.
[0040] In the following, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more. In order to facilitate the understanding of the above-mentioned technical solutions and the technical effects produced by the embodiments of the present application, the embodiments of the present application first explain the relevant professional terms:
[0041] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively and comprehensively applies advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.
[0042] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), referred to as IVICS, is a development direction of Intelligent Transportation Systems (ITS). IVICS uses advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and conducts active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.
[0043] Computer vision is a simulation of biological vision using computers and related equipment. Its main task is to obtain three-dimensional information of the corresponding scene by processing the collected images or videos, just like humans and many other creatures do every day.
[0044] Artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0045] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0046] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0047] Optical Character Recognition: OCR. It refers to the process of determining the shape of characters printed by electronic devices by detecting dark and light patterns, and then translating the shape into computer text using character recognition methods, as well as the technology of converting the text in the image into text format through recognition software for further editing and processing by word processing software.
[0048] N-face retrieval: Find one or more faces with the highest similarity to the face to be retrieved in a large-scale face database. The retrieval performance is related to the size of the database N.
[0049] Microservice: A software development technology, a variant of the service-oriented architecture style, which advocates dividing a single application into a group of small services, which coordinate and cooperate with each other to provide ultimate value to users. Each service runs in its own independent process, and services communicate with each other using lightweight communication mechanisms. Each service is built around a specific business and can be independently deployed to a production environment, a production-like environment, etc. In addition, a unified and centralized service management mechanism should be avoided as much as possible. For a specific service, the appropriate language and tools should be selected according to the context to build it.
[0050] In related technologies, with the rapid development of the Internet, more and more people are participating in the Internet, and the scale of public network content has doubled, but the quality of the corresponding content is uneven. In order to ensure the clean and healthy development of the Internet, it is necessary to review the content of the public cloud, so as to timely process and filter the information that is not suitable for dissemination on the public network. This objective demand has led to the emergence of many visual tasks, but due to the limitation of related technologies, there is no mature multi-visual task collaborative scheduling framework, and the implementation and scheduling of multi-visual tasks requires different degrees of human participation.
[0051] The more common public cloud content review is usually conducted manually, which requires content review specialists to analyze and repeatedly confirm massive amounts of images and videos, resulting in high labor costs. The review is also required to be online for a long time, whether at night or on holidays, and long processing times will cause fatigue and overload of reviewers. In addition, the manual review method adopted by this public cloud image review requires review specialists to observe with the naked eye, and there is an upper limit to the processing capacity. It takes at least 3 hours to manually review 10,000 images, resulting in low review efficiency. Furthermore, the manual review method requires review specialists to observe with the naked eye and conduct random inspections, which places high demands on personnel, and human errors and omissions are inevitable.
[0052] In order to improve the operating efficiency of visual tasks, get rid of dependence on artificial vision, and enable multiple visual tasks to be rationally and automatically scheduled according to the relationship between each other, an embodiment of the present application provides an image processing method, which can provide a framework for collaborative scheduling of multiple visual tasks. Each visual task running within the framework is based on a linear communication model, and the subtasks in each visual task are encapsulated in the form of microservices. The same subtasks between different visual tasks can be reused, so that the services associated with each visual task can be reasonably utilized to ensure the collaborative and efficient operation of multiple visual tasks. Under this framework, the system scheduling of multiple visual tasks no longer relies on human intervention, and can be implemented fully automatically and efficiently, significantly improving the image processing efficiency.
[0053] The embodiments of the present application do not limit the application scenarios to which the processed images are adapted. For example, image processing can be performed in scenarios such as cloud gaming, cloud communities, cloud e-commerce, and smart driving.
[0054] The embodiments of the present application may involve cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology can be used to build specific scenarios where multiple servers are located. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the application of cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, image websites, and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.
[0055] The embodiments of the present application can be applied to various cloud applications provided based on cloud technology. For example, medical cloud. Medical cloud refers to the use of "cloud computing" to create a medical and health service cloud platform based on new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the application and combination of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates residents' medical treatment. For example, hospital appointment registration, medical insurance, etc. are all the products of the combination of cloud computing and the medical field. The medical cloud also has the advantages of data security, information sharing, dynamic expansion, and overall layout.
[0056] Another example is cloud IoT. The Internet of Things (IoT) refers to the real-time collection of any object or process that needs attention, connection, and interaction through various devices and technologies such as information sensors, radio frequency identification technology, global positioning system, infrared sensor, laser scanner, etc., and the collection of various required information such as sound, light, heat, electricity, mechanics, chemistry, biology, location, etc., and the realization of ubiquitous connection between objects and objects, objects and people through various possible network access, and the realization of intelligent perception, identification and management of objects and processes. The Internet of Things is an information carrier based on the Internet, traditional telecommunications networks, etc. It allows all ordinary physical objects that can be independently addressed to form an interconnected network.
[0057] Another example is cloud calling. Cloud Call Center is a call center system built based on cloud computing technology. Enterprises do not need to purchase any software or hardware systems. They only need to have basic conditions such as personnel and venues to quickly own their own call center. The software and hardware platform, communication resources, daily maintenance and services are provided by the server vendor. It has many characteristics such as short construction period, low investment, low risk, flexible deployment, strong system capacity scalability, and low operation and maintenance costs. Whether it is a telephone marketing center or a user service center, enterprises only need to rent services on demand to establish a comprehensive, stable, reliable call center system with seats distributed all over the country and nationwide call access.
[0058] Another example is private cloud, public cloud or hybrid cloud. Private cloud is to create cloud infrastructure and software and hardware resources within the firewall so that departments within an organization or enterprise can share resources in the data center. In addition to hardware resources, cloud equipment (IaaS, Infrastructure as a Service) software is generally required to create a private cloud. Public cloud usually refers to a cloud that a third-party provider provides to users. Public cloud is generally available through the Internet and may be free or low-cost. The core attribute of public cloud is shared resource services. There are many instances of this cloud that can provide services throughout today's open public network. Hybrid cloud combines public cloud and private cloud, and is the main model and development direction of cloud computing in recent years.
[0059] Another example is cloud gaming. Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games. In a cloud gaming scenario, the game is not played on the player's game terminal, but on a cloud server, which renders the game scene into a video and audio stream and transmits it to the player's game terminal over the network. The player's game terminal does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input commands and send them to the cloud server.
[0060] Another example is cloud conferencing. Cloud conferencing is an efficient, convenient, and low-cost form of conferencing based on cloud computing technology. Users only need to use the Internet interface and perform simple and easy-to-use operations to quickly and efficiently share voice, data files, and videos with teams and users around the world. The cloud conferencing service provider helps users operate complex technologies such as data transmission and processing during the meeting.
[0061] The method provided in the embodiment of the present application may also involve blockchain, that is, the method provided in the embodiment of the present application may be implemented based on blockchain, or the data involved in the method provided in the embodiment of the present application may be stored based on blockchain, or the execution subject of the method provided in the embodiment of the present application may be located in blockchain. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.
[0062] The underlying blockchain platform can include processing modules such as user management, basic services, and smart contracts. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining the generation of public and private keys (account management), key management, and the maintenance of the corresponding relationship between the user's real identity and the blockchain address (authority management), etc., and under authorization, supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records valid requests to storage after consensus is reached. For a new business request, the basic service first adapts the interface for parsing and authentication processing (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management), and transmits it to the shared ledger completely and consistently after encryption (network communication), and records and stores it; the smart contract module is responsible for the registration and issuance of contracts, as well as contract triggering and contract execution. Developers can define the contract logic in a certain programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, call the key or other events to trigger the execution and complete the contract logic. At the same time, it also provides the function of contract upgrade and cancellation.
[0063] The platform product service layer provides the basic capabilities and implementation framework of typical applications. Developers can superimpose business features based on these basic capabilities to complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for business participants to use.
[0064] See also Figure 1 , Figure 1 is a schematic diagram of a feasible implementation framework of the image processing method provided in the embodiments of this specification, such as Figure 1 As shown, the implementation framework may include at least a business layer component 01, an access layer component 02, a microservice cluster 03, and a model layer component 04. Among them, the above framework of the embodiment of the present invention can provide the image processing capabilities required by applications in various scenarios, including but not limited to cloud technology, cloud games, cloud rendering, artificial intelligence, smart transportation, assisted driving, video media, smart communities, instant messaging, etc. Each component in the framework can be a terminal device or a server. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc.
[0065] Specifically, the business layer component 01 is used to obtain the above-mentioned image. The access layer component 02 is used to set the configuration data for the above-mentioned image, and the above-mentioned configuration data is used to limit at least one first visual task for processing the above-mentioned image, and configure the linear communication model corresponding to each first visual task. Any of the above-mentioned first visual tasks is a linear communication task including at least one subtask, and each subtask points to a corresponding microservice, and the above-mentioned microservice runs based on the corresponding visual processing model. Microservice cluster 03 is used to manage each microservice. The model layer component is used to manage each visual processing model. In addition, the above-mentioned access layer component is also used to trigger the operation of the second visual task, so that the above-mentioned second visual task determines the first subtask according to the corresponding linear communication model, calls the microservice corresponding to the above-mentioned first subtask, so that the microservice corresponding to the above-mentioned first subtask processes the above-mentioned image and obtains the first processing result corresponding to the above-mentioned first subtask. The above-mentioned second visual task is any first visual task in the above-mentioned at least one first visual task.
[0066] In one embodiment, please refer to Figure 2 , which shows another feasible implementation framework schematic diagram of the image processing method provided in the embodiment of this specification. Figure 1 It is further optimized on the basis of the previous one, providing a more refined division of functional components, making the functional components more decoupled from each other, forming a refined, hierarchical, multi-interactive, and low-coupled image processing framework.
[0067] The business layer can be composed of business layer component 01, which mainly completes relevant user access, user capability configuration and image processing result display. It also includes operation platform, control background and front end, among which the operation platform is responsible for completing relevant user access, the control background is responsible for completing user capability configuration. The front end is responsible for completing image processing result display.
[0068] The routing layer is mainly responsible for the routing and load balancing of related requests to ensure high availability of the system. The routing layer mainly includes three modules: access layer routing, internal microservice routing and external microservice routing.
[0069] The flow limiting layer mainly limits the frequency at the user level to prevent the system from overloading due to special user traffic, thereby ensuring the efficiency and responsiveness of image processing. The flow limiting layer mainly includes user frequency limiting management and frequency limiting services.
[0070] The access layer can be composed of access layer component 02. The access layer is composed of 3 sub-modules. Among them, configuration management and degradation strategy are used to configure specific parameters that provide image processing capabilities. In the embodiment of the present application, the subtasks of the visual task are all encapsulated into microservices, and the microservice runs by calling the corresponding model, and the access layer can also manage the model. Each visual task can be managed in the access layer. The embodiment of the present application does not limit the specific content of the visual task, and the corresponding visual task can be constructed according to the specific application situation. For example, detecting text and images in an image can generate text and images visual tasks, and detecting advertisements in an image can generate advertisement visual tasks. The visual tasks in the embodiment of the present application are all linear communication tasks, and linear communication tasks are represented by PPL (Pipeline). The DAG module in the access layer is used to store the logical structure corresponding to the model, and the expression module is used to store the decision strategy, pre-processing strategy and post-processing strategy of the model, which are used to configure the processing methods before and after the image is input into the model, respectively, wherein DAG is Database Availability Group, representing a directed acyclic graph.
[0071] In the embodiment of the present application, the subtasks in each visual task are encapsulated as microservices, and the microservices are uniformly provided by a microservice cluster. The microservices corresponding to the subtasks in different visual tasks can be reused. The microservices shield the underlying model and expose simple and easy-to-use interfaces to facilitate quick calls. The microservice layer is the key to CV calculations.
[0072] During the operation of the image processing framework, you can also perform fine operation and maintenance operations on it, such as controlling the deployed services, scaling their capacity, sensing related indicators, or downgrading services according to the response situation. In addition, the generated log information is recorded in a timely manner during the operation of the image processing framework, and relevant retrieval is performed based on the processing results obtained by the image processing framework for the image.
[0073] The model layer in the embodiment of the present application stores the CV models that each microservice in the microservice cluster needs to call. The basic components in the embodiment of the present application are used to provide basic support for the upper-level visual tasks, such as Mysql that provides data storage support, Redis that provides data storage support, or cloud storage service components. Mysql is a relational database management system, and Redis (Remote Dictionary Server) is a remote dictionary service.
[0074] Please refer to Figure 3 , which shows that based on Figure 2 The data processing flow chart of the framework. The user uploads the media resources to the image processing framework. The embodiment of the present application does not limit the media resources, which can be images, videos or video streams, and the media resources can be uploaded through local storage, live broadcast, etc. The image processing framework operation background can set a policy for configuring the media resources through the security operation configuration, and the policy acts on the media resources through user extension services. Developers can configure the specific parameters required for the execution of the above policies based on the R&D background corresponding to the image processing framework. The parameter can be model service information or current limiting rules, and then the background configuration can be automatically deployed to the model background. The image processing gateway can schedule and trigger the various visual tasks corresponding to the media resources based on the configuration of the operation background and the R&D background, and obtain the final processing results.
[0075] An image processing method according to an embodiment of the present application is introduced below. The information involved in the present disclosure may be information authorized by the user or fully authorized by all parties.
[0076] Figure 4 A flow chart of an image processing method provided by an embodiment of the present application is shown. The embodiment of the present application provides method operation steps as described in the embodiment or flow chart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system, terminal device or server product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiment or the accompanying drawings. The above method may include:
[0077] S101. Acquire an image and configuration data for the image, wherein the configuration data is used to define at least one first visual task for processing the image, wherein any of the first visual tasks is a linear communication task including at least one subtask, each subtask points to a corresponding microservice, and the microservice runs based on a corresponding visual processing model.
[0078] For the media data received by the image processing framework, an image can be obtained based on the media data, thereby triggering at least one first visual task performed on the image, wherein the configuration data can come from the operation background or R&D background mentioned above. The embodiment of the present application does not limit the number of first visual tasks, that is, the image processing framework can support the operation of a single visual task, and can also support the coordinated operation of multiple visual tasks.
[0079] The application embodiment does not limit the specific type of the first visual task. For example, the first visual task can classify the image, that is, determine whether the image belongs to a certain specific type, such as whether the image belongs to an advertisement, or whether the image belongs to an image that spreads bad information. Or the first visual task can also detect specific information in the image, such as detecting whether there is a preset text or a preset pattern in the image. The first visual task can also be constructed according to specific image processing requirements. For example, detect whether a certain event has occurred, whether it belongs to a certain specific scene, etc.
[0080] S102. Determine a first subtask according to a linear communication model corresponding to the second visual task, and call a microservice corresponding to the first subtask, so that the microservice corresponding to the first subtask processes the image to obtain a first processing result corresponding to the first subtask, wherein the second visual task is any first visual task among the at least one first visual task.
[0081] The first subtask is the first task in the second visual task, so the input data of the first subtask is the image acquired in step S101. For example, three first visual tasks are determined in step S101, namely visual task 1 and visual task 2. Visual task 1 includes three subtasks, namely feature extraction task, face detection task and face classification task. Visual task 2 includes two tasks, namely feature extraction task and image and text recognition task. Taking visual task 1 as the second visual task as an example, the first subtask is the feature extraction task corresponding to visual task 1. Step S102 can call the microservice corresponding to the feature extraction task corresponding to visual task 1, and the microservice runs by calling the feature extraction model.
[0082] In one embodiment, in response to the completion of the execution of the above-mentioned first subtask, a second subtask can be determined, where the above-mentioned second subtask is a subtask determined based on the linear communication model corresponding to the above-mentioned second visual task and is located after the above-mentioned first subtask; the microservice corresponding to the above-mentioned second subtask is called so that the microservice corresponding to the above-mentioned second subtask processes the above-mentioned image or the above-mentioned first processing result to obtain the second processing result corresponding to the above-mentioned second subtask.
[0083] Still taking visual task 1 as the second visual task as an example, after successfully executing the feature extraction task, the face detection task can be determined as the above-mentioned second subtask, thereby triggering the execution of each subtask in the second visual task in sequence until the second visual task is completed. No human intervention is required in the process, that is, it can be executed fully automatically after the linear communication model is determined.
[0084] In one embodiment, the method further includes: in response to the completion of the execution of the first subtask, determining a third visual task, wherein the third visual task is a visual task among the at least one first visual task, includes the first subtask, and is different from the second visual task; determining a third subtask, wherein the third subtask is a subtask determined based on a linear communication model corresponding to the third visual task and is located after the first subtask; calling a microservice corresponding to the third subtask, so that the microservice corresponding to the third subtask processes the image or the first processing result to obtain a third processing result corresponding to the third subtask.
[0085] Still taking visual task 1 as the second visual task as an example, after successfully executing the feature extraction task, since visual task 2 also includes the feature extraction task, the running of the image and text recognition task after the feature extraction task in visual task 2 can be triggered until visual task 2 is completed. This method can execute the subtasks common to each visual task only once, and trigger the running of other related visual tasks based on the execution results, and reuse the above execution results during the running process, thereby improving the scheduling efficiency and running efficiency of multiple visual tasks.
[0086] In one embodiment, the scheduling method of a first visual task can also be determined according to the configuration data. Figure 5 , which shows a scheduling method of a certain first visual task is a schematic diagram, and the scheduling method is obtained based on the linear communication model of the certain first visual model. For example, on the basis of acquiring the image, image pre-processing can be performed, and then subtask 1, subtask 2 and subtask 3 are scheduled in parallel, and subtask 1, subtask 2 and subtask 3 correspond to model 1, model 2 and model 3 respectively. After integrating the processing results of the above three subtasks, subtask 4 and subtask 5 are scheduled in turn, and subtask 4 and subtask 5 correspond to model 4 and model 5 respectively. The output result of the certain first visual task is obtained according to the scheduling results of each subtask, and the output result can be represented by the classification score of the image.
[0087] In one embodiment, the linear communication model corresponding to the second visual task also includes a decision strategy for the output results of each subtask, and the method also includes: in response to the second visual task being completed, processing the processing results output by each subtask in the second visual task based on the decision strategy to obtain the execution result of the second visual task.
[0088] In a feasible implementation, the above decision strategy can be expressed by an expression. Specifically, the expression can be a numerical expression, which is used to represent the logic of combining the results obtained by each subtask involved in the visual task. For example, if the expression is ExprA if Cond else ExprB, then if Cond is true, ExprA is returned, otherwise, ExprB is returned. Of course, the expression can also include four arithmetic operations or Boolean expressions. The priority of multiplication and division of four arithmetic operations is higher than that of addition and subtraction. Boolean expression ExprA (>|>=|<|<=) ExprB: Compare ExprA and ExprB and output the corresponding result.
[0089] In one embodiment, the method further includes: in response to the failure of the execution of the second visual task, determining the fourth subtask that failed to execute; based on the linear communication model corresponding to the second visual task, extracting the processing results corresponding to the subtasks executed before the fourth subtask in the second visual task; and resuming the execution of the fourth subtask based on the extracted processing results.
[0090] Please refer to Figure 6 , which shows a schematic diagram of another scheduling method of a first visual task, which is represented by a directed acyclic graph, and the subtasks in the other first visual task include decoding. Detection 1, detection 2, classification, registration and five-point face detection. If a subtask fails to execute, the failed subtask can be restored according to the results of the directed acyclic graph and the output of other subtasks before, thereby increasing the probability of successful scheduling of the second visual task.
[0091] In one embodiment, the target microservice corresponds to at least two visual processing models, and the target microservice is any one of the tasks in the at least one visual task. The configuration data is also used to limit the target visual processing model called by the microservice corresponding to the target task, and the target visual processing model is any one of the at least two visual processing models.
[0092] Specifically, for the target microservice, you can configure a single model, a combined model, or a fusion model. The execution effects of the single model, the fusion model, and the combined model are enhanced in turn, but the time consumption and resource usage rate increase in turn. Therefore, you can determine the specific model to be selected based on the specific configuration data. Please refer to Figure 7 , which shows a schematic diagram of the model distribution corresponding to the microservice. Taking the visual task of human body classification as an example, the single model only includes one human body classification model, and the fusion model includes three human body classification models. The output results of the above three human body classification models can be fused to improve the detection accuracy. The combined model is not only configured with a human body classification model, but also can perform human body part detection, filtering, cutting out, and infilling, thereby reducing image noise, improving the detection accuracy of the human body classification model, and further improving the detection accuracy.
[0093] In one embodiment, the model corresponding to the target microservice can be managed: for a single model, the score conversion table corresponding to the model output label is specified; for a fusion model, the atomic model used, the score table of the atomic model, and the final score conversion table are specified; for a combination model, the atomic model used, the score table of the atomic model, and the combination capability expression are specified.
[0094] Before executing each visual task, the acquired image can be pre-processed, that is, the image preprocessing process is executed. The preprocessing process includes encoding and decoding, GIF image frame extraction, long image segmentation and picture-in-picture detection, and multiple sub-images are obtained from the input original image. Please refer to Figure 8 , which shows a schematic diagram of image preprocessing.
[0095] for Figure 8 In the sampling link, if the sampling identifier frame_sample_param does not exist in the configuration data, the sampling logic of the gif image will not be triggered. At this time, by default, only the first frame of the gif image is obtained for visual task processing. If there is an identifier representing random sampling in the configuration data, the max_frame_num frames are randomly taken out from all the frames of the gif image for processing. When max_frame_num> the total number of frames of the gif image, all frames are extracted for processing, where max_frame_num is the number of samples set in the configuration data. If there is an identifier representing fixed-interval sampling in the configuration data, fixed-interval sampling is performed according to the sampling interval in the configuration data. The fixed interval is identified by min_sample_interval. If max_frame_num>0 and the number of samples>max_frame_num, the sampling interval is forced to be expanded so that the number of samples = max_frame_num. Of course, in some special scenarios, all frames can be collected.
[0096] for Figure 8 In the segmentation process, if the segmentation flag split_image_param does not exist in the configuration data, the long image segmentation logic will not be triggered; if the long image segmentation parameter max_aspect_ratio is set in the configuration data, and the long side / short side of the input image is greater than the max_aspect_ratio, the long image segmentation process is performed; after segmentation, the width and height of each sub-image are exactly the same, the short side length of the sub-image is the same as the short side length of the image, and the long side length of the sub-image is equal to the short side length of the sub-image multiplied by the first preset parameter expect_aspect_ratio; the long side overlap ratio of two adjacent sub-images is not less than the second preset parameter min_overlap_ratio; the union of all sub-images can completely cover the original image, and the first preset parameter expect_aspect_ratio and the second preset parameter min_overlap_ratio are both recorded in the configuration data.
[0097] In the image preprocessing stage, the preprocessing operations used in the image preprocessing stage can also be controlled according to actual conditions, such as Fig. 9 As shown, it shows a schematic diagram of the selection of preprocessing operations in the preprocessing stage. Among them, N0, N1, and N2 indicate the number of frames of the gif, and the first-level filtering, second-level filtering, and third-level filtering are the strategies that can be selected in the preprocessing stage, and the filtering degree increases in turn. The greater the filtering degree, the fewer the number of images output by the preprocessing stage.
[0098] like Fig.10 As shown in the figure, it shows a schematic diagram of the degradation operation in the preprocessing stage. Lossless degradation can also be achieved by adjusting the request QPS, etc., and slight lossy degradation can be achieved by reducing the number of images output by preprocessing, timeout rejection, etc.; severe lossy degradation can be achieved by closing capabilities, etc. to ensure that the system runs stably under ultra-high traffic. The capabilities are provided by opening visual tasks through the image processing framework. Among them, QPS represents the query rate per second (Queries-per-second), which is a measure of the amount of traffic processed by a specific query server within a specified time.
[0099] Specifically, lossless degradation can be achieved by reducing existing tasks, reducing QPS, and adjusting capacity. In the stage of mild degradation, mild degradation can be achieved by adjusting the image pre-processing strategy, selecting a model with faster response among multiple models, or rejecting excessive QPS requests as soon as possible. In the stage of severe degradation, only the capacity supply of important users is guaranteed.
[0100] After executing each visual task, the acquired image can also be post-processed. The post-processing can fuse the output results of each visual task to obtain a new processing result. For example, the output result of one visual task represents the presence of text and pictures in the image, and the output result of another visual task represents the presence of a station logo in the image. Then, the new processing result represents that the image is a TV program from the TV station corresponding to the station logo.
[0101] The embodiments of the present application do not limit the specific content of the visual task. Taking the above-mentioned second visual task as a face detection task as an example, the above-mentioned face detection task includes a decoding subtask, a face position detection task and a face type recognition task. The above-mentioned decoding subtask is used to decode the above-mentioned image to obtain decoding data for inputting the above-mentioned face position detection task. The above-mentioned face position detection task is used to determine the face position based on the above-mentioned decoding data. The above-mentioned face type recognition task is used to identify the face type of the face at the above-mentioned face position. The above-mentioned face type includes a real face, a cartoon face or a face belonging to a face set.
[0102] like Fig.11 As shown, it shows a schematic diagram of the face detection task framework. First, the image is decoded to obtain decoded data, and then face detection is performed based on the decoded data. In the face detection, based on the face position, it can be identified whether the face at the position has five-point features, comic features and IP features, whether the face with five-point features belongs to a real person in the real person library, whether the face has comic features, whether it belongs to a comic in the comic library, and whether the face is a face with intellectual property (IP) protection.
[0103] like Fig.12 As shown in the figure, it shows a schematic diagram of the face detection task process. For the input image, face detection is first performed, and face detection will output the face frame and face frame type. If there is no face frame, it may be because the image selection causes the detection to fail, so it is necessary to introduce four-angle rotation detection; then four angles (0, 90, 180, 270) are classified based on the face frame to obtain the face frame rotation angle; according to the face frame type, if it is a comic frame, comic feature retrieval is performed; if it is a real person frame, five-point feature and retrieval are performed after angle correction to complete the real person retrieval process.
[0104] In one embodiment, before executing the fifth subtask, the current resource occupancy status is obtained, and the fifth subtask is any subtask in the second visual task; according to the resource occupancy requirement of the fifth subtask and the resource occupancy status, the target resource for processing the fifth subtask is determined, and the target resource is the running resource of the microservice corresponding to the fifth subtask, and the running resource is a central processing unit resource or a graphics processing unit resource.
[0105] Specifically, if Fig.13 As shown, it shows a schematic diagram of resource scheduling. The left side is a schematic diagram of the scheduling result of running all subtasks in the GPU without considering the current resource occupancy and the specific resource occupancy requirements of the subtasks. The right side takes into account the current resource occupancy status and the specific resource occupancy requirements of the subtasks, and evenly schedules this task, with some subtasks running in the GPU and some subtasks running in the CPU. By introducing the idea of instruction pipeline for CPU and GPU heterogeneous computing, the CPU or GPU is avoided from being overloaded or idle, and a visual task is split into multiple subtasks. Each subtask uses the CPU or GPU for calculation in a differentiated manner; different visual tasks are scheduled in parallel to make full use of computing resources. Among them, GPU refers to Graphics Processing Unit. CPU refers to Central Processing Unit.
[0106] In the image processing method provided by this application, each microservice is implemented by calling at least one corresponding model, and each model is an AI model configured by the embodiment of this application to perform visual tasks. The method provided by this application integrates many AI models, which can support efficient processing of massive images and greatly reduce labor costs. The current QPS is more than 10,000, which means that the processing of tens of thousands of images can be completed in seconds, which greatly improves the efficiency of image processing and is conducive to large-scale promotion. The integrated AI model adopts a number of advanced technologies, such as face and body recognition, motion detection, OCR, etc., which greatly ensures the reliability of image processing results, and its algorithm effect can achieve industry-leading level.
[0107] Please refer to Fig.14 , which shows a block diagram of an image processing device in this embodiment, the device includes:
[0108] A data acquisition module 101 is used to acquire an image and configuration data for the image, wherein the configuration data is used to define at least one first visual task for processing the image, wherein any of the first visual tasks is a linear communication task including at least one subtask, each subtask pointing to a corresponding microservice, and the microservice runs based on a corresponding visual processing model;
[0109] The data processing module 102 is used to determine the first subtask according to the linear communication model corresponding to the second visual task, and call the microservice corresponding to the above-mentioned first subtask, so that the microservice corresponding to the above-mentioned first subtask processes the above-mentioned image to obtain the first processing result corresponding to the above-mentioned first subtask. The above-mentioned second visual task is any first visual task among the above-mentioned at least one first visual task.
[0110] In one embodiment, the data processing module is used to perform the following operations:
[0111] In response to the completion of the first subtask, determining a second subtask, wherein the second subtask is a subtask determined based on the linear communication model corresponding to the second visual task and is located after the first subtask;
[0112] The microservice corresponding to the second subtask is called, so that the microservice corresponding to the second subtask processes the image or the first processing result to obtain the second processing result corresponding to the second subtask.
[0113] In one embodiment, the data processing module is used to perform the following operations:
[0114] In response to the completion of the first subtask, determining a third visual task, where the third visual task is a visual task in the at least one first visual task, includes the first subtask, and is different from the second visual task;
[0115] Determine a third subtask, where the third subtask is determined based on the linear communication model corresponding to the third visual task and is located after the first subtask;
[0116] The microservice corresponding to the third subtask is called, so that the microservice corresponding to the third subtask processes the image or the first processing result to obtain a third processing result corresponding to the third subtask.
[0117] In one embodiment, the linear communication model corresponding to the second visual task further includes a decision strategy for the output results of each subtask, and the data processing module is used to perform the following operations:
[0118] In response to the second visual task being completed, the processing results output by each subtask in the second visual task are processed based on the decision strategy to obtain the execution result of the second visual task.
[0119] In one embodiment, the data processing module is used to perform the following operations:
[0120] In response to the second visual task failing to be executed, determining a fourth subtask that failed to be executed;
[0121] Based on the linear communication model corresponding to the second visual task, extracting processing results corresponding to each subtask executed before the fourth subtask in the second visual task;
[0122] Based on the extracted processing results, the fourth subtask is resumed.
[0123] In one embodiment, the data processing module is used to perform the following operations:
[0124] Before executing the fifth subtask, obtaining the current resource occupancy state, the fifth subtask being any subtask in the second visual task;
[0125] According to the resource occupancy requirement of the fifth subtask and the resource occupancy status, a target resource for processing the fifth subtask is determined, the target resource is an operating resource of the microservice corresponding to the fifth subtask, and the operating resource is a central processing unit resource or a graphics processing unit resource.
[0126] In one embodiment, the target microservice corresponds to at least two visual processing models, and the target microservice is any one of the tasks in the at least one visual task. The configuration data is also used to limit the target visual processing model called by the microservice corresponding to the target task, and the target visual processing model is any one of the at least two visual processing models.
[0127] In one embodiment, the second visual task is a face detection task, which includes a decoding subtask, a face position detection task and a face type recognition task. The decoding subtask is used to decode the image to obtain decoding data for inputting the face position detection task. The face position detection task is used to determine the face position based on the decoding data. The face type recognition task is used to identify the face type of the face at the face position. The face type includes a real face, a cartoon face or a face belonging to a face set.
[0128] The device embodiment of the present application and the method embodiment described above are based on the same inventive concept and will not be described in detail here.
[0129] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the above-mentioned image processing method.
[0130] The embodiment of the present application further provides a computer-readable storage medium, which can store a plurality of instructions. The instructions can be suitable for a processor to load and execute the image processing method described above in the embodiment of the present application.
[0131] In one embodiment, the above-mentioned image processing method includes:
[0132] Acquire an image and configuration data for the image, the configuration data being used to define at least one first visual task for processing the image, wherein any of the first visual tasks is a linear communication task including at least one subtask, each subtask pointing to a corresponding microservice, and the microservice running based on a corresponding visual processing model;
[0133] According to the linear communication model corresponding to the second visual task, the first subtask is determined, and the microservice corresponding to the first subtask is called so that the microservice corresponding to the first subtask processes the image to obtain the first processing result corresponding to the first subtask. The second visual task is any first visual task among the at least one first visual task.
[0134] In one embodiment, the method further includes:
[0135] In response to the completion of the first subtask, determining a second subtask, wherein the second subtask is a subtask determined based on the linear communication model corresponding to the second visual task and is located after the first subtask;
[0136] The microservice corresponding to the second subtask is called, so that the microservice corresponding to the second subtask processes the image or the first processing result to obtain the second processing result corresponding to the second subtask.
[0137] In one embodiment, the method further includes:
[0138] In response to the completion of the first subtask, determining a third visual task, where the third visual task is a visual task in the at least one first visual task, includes the first subtask, and is different from the second visual task;
[0139] Determine a third subtask, where the third subtask is determined based on the linear communication model corresponding to the third visual task and is located after the first subtask;
[0140] The microservice corresponding to the third subtask is called, so that the microservice corresponding to the third subtask processes the image or the first processing result to obtain a third processing result corresponding to the third subtask.
[0141] In one embodiment, the linear communication model corresponding to the second visual task further includes a decision strategy for the output results of each subtask, and the method further includes:
[0142] In response to the second visual task being completed, the processing results output by each subtask in the second visual task are processed based on the decision strategy to obtain the execution result of the second visual task.
[0143] In one embodiment, the method further includes:
[0144] In response to the second visual task failing to be executed, determining a fourth subtask that failed to be executed;
[0145] Based on the linear communication model corresponding to the second visual task, extracting processing results corresponding to each subtask executed before the fourth subtask in the second visual task;
[0146] Based on the extracted processing results, the fourth subtask is resumed.
[0147] In one embodiment, the method further includes:
[0148] Before executing the fifth subtask, obtaining the current resource occupancy state, the fifth subtask being any subtask in the second visual task;
[0149] According to the resource occupancy requirement of the fifth subtask and the resource occupancy status, a target resource for processing the fifth subtask is determined, the target resource is an operating resource of the microservice corresponding to the fifth subtask, and the operating resource is a central processing unit resource or a graphics processing unit resource.
[0150] In one embodiment, the target microservice corresponds to at least two visual processing models, and the target microservice is any one of the tasks in the at least one visual task. The configuration data is also used to limit the target visual processing model called by the microservice corresponding to the target task, and the target visual processing model is any one of the at least two visual processing models.
[0151] In one embodiment, the second visual task is a face detection task, which includes a decoding subtask, a face position detection task and a face type recognition task. The decoding subtask is used to decode the image to obtain decoding data for inputting the face position detection task. The face position detection task is used to determine the face position based on the decoding data. The face type recognition task is used to identify the face type of the face at the face position. The face type includes a real face, a cartoon face or a face belonging to a face set.
[0152] Further, Fig.15 A schematic diagram of the hardware structure of a device for implementing the method provided in the embodiment of the present application is shown. The above device may participate in constituting or include the apparatus or system provided in the embodiment of the present application. Fig.15As shown, the device 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.15 The structure shown is for illustration only and does not limit the structure of the electronic device. Fig.15 More or fewer components as shown, or with Fig.15 Different configurations are shown.
[0153] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0154] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the above-mentioned method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned image processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0155] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the device 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.
[0156] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of device 10 (or mobile device).
[0157] Fig.16 1 is a block diagram of another electronic device according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Fig.16 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.
[0158] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.
[0159] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.
[0160] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of the embodiments of the present application are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0161] Each embodiment in the embodiments of the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0162] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program. The above program may be stored in a computer-readable storage medium, and the above storage medium may be a read-only memory, a disk or an optical disk, etc.
[0163] The above is only a preferred embodiment of the embodiment of the present application and is not intended to limit the embodiment of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiment of the present application should be included in the protection scope of the embodiment of the present application.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire an image and configuration data for the image, wherein the configuration data is used to define at least one first visual task for processing the image, wherein any of the first visual tasks is a linear communication task including at least one subtask, each subtask points to a corresponding microservice, and the microservice corresponds to three target visual processing models: a single model, a combined model, or a fusion model. The configuration data defines the target visual processing model called by the microservice, specifies a score conversion table corresponding to an output label for a single model, specifies an atomic model to be used, a score table of the atomic model, and a final score conversion table for a fusion model; and specifies an atomic model to be used, a score table of the atomic model, and a combination capability expression for a combined model; According to the linear communication model corresponding to the second visual task, a first subtask is determined, and a microservice corresponding to the first subtask is called, so that the microservice corresponding to the first subtask processes the image to obtain a corresponding first processing result, and the second visual task is any of the first visual tasks; when the second visual task is completed, the output results of each subtask of the second visual task are processed based on a numerical expression to obtain an output result of the second visual task, and the numerical expression includes four arithmetic operations or a Boolean expression to express the logic of combining the output results of each subtask; When the first subtask is completed, a third visual task is determined, which is any first visual task that includes the first subtask and is different from the second visual task; a third subtask that is located after the first subtask in the third visual task is determined; and a microservice corresponding to the third subtask is called to obtain a processing result corresponding to the third subtask; The output results of the second visual task and the third visual task are integrated to obtain a new processing result.
2. The method according to claim 1, characterized in that The method further comprises: In response to the completion of the first subtask, determining a second subtask, where the second subtask is determined based on the linear communication model corresponding to the second visual task and is located after the first subtask; The microservice corresponding to the second subtask is called, so that the microservice corresponding to the second subtask processes the image or the first processing result to obtain a second processing result corresponding to the second subtask.
3. The method according to claim 1, characterized in that The method further comprises: In response to the second visual task failing to be executed, determining a fourth subtask that failed to be executed; Extracting, based on the linear communication model corresponding to the second visual task, processing results corresponding to each subtask executed before the fourth subtask in the second visual task; Based on the extracted processing results, the fourth subtask is resumed.
4. The method according to claim 1, characterized in that: The method further comprises: Before executing a fifth subtask, obtaining a current resource occupancy state, wherein the fifth subtask is any subtask in the second visual task; According to the resource occupancy requirement of the fifth subtask and the resource occupancy status, a target resource for processing the fifth subtask is determined, the target resource is an operating resource of the microservice corresponding to the fifth subtask, and the operating resource is a central processing unit resource or a graphics processing unit resource.
5. The method according to claim 1, characterized in that The second visual task is a face detection task, which includes a decoding subtask, a face position detection task and a face type recognition task. The decoding subtask is used to decode the image to obtain decoding data for inputting the face position detection task. The face position detection task is used to determine the face position based on the decoding data. The face type recognition task is used to identify the face type of the face at the face position. The face type includes a real face, a cartoon face or a face belonging to a face set.
6. An image processing device, characterized in that: The device comprises: A data acquisition module, used to acquire an image and configuration data for the image, wherein the configuration data is used to define at least one first visual task for processing the image, wherein any of the first visual tasks is a linear communication task including at least one subtask, wherein each subtask points to a corresponding microservice, wherein the microservice corresponds to three target visual processing models, namely, a single model, a combined model, or a fusion model, wherein the configuration data defines the target visual processing model called by the microservice, and specifies a score conversion table corresponding to an output label for a single model, specifies an atomic model to be used, a score table of the atomic model, and a final score conversion table for a fusion model; and specifies an atomic model to be used, a score table of the atomic model, and a combination capability expression for a combined model; a data processing module, configured to determine a first subtask according to a linear communication model corresponding to a second visual task, call a microservice corresponding to the first subtask, so that the microservice corresponding to the first subtask processes the image to obtain a corresponding first processing result, wherein the second visual task is any of the first visual tasks; and when the second visual task is completed, process the output results of each subtask of the second visual task based on a numerical expression to obtain an output result of the second visual task, wherein the numerical expression includes four arithmetic operations or a Boolean expression to express the logic of combining the output results of each subtask; When the first subtask is completed, a third visual task is determined, which is any first visual task that includes the first subtask and is different from the second visual task; a third subtask that is located after the first subtask in the third visual task is determined; and a microservice corresponding to the third subtask is called to obtain a processing result corresponding to the third subtask; The output results of the second visual task and the third visual task are integrated to obtain a new processing result.
7. An image processing system, characterized in that: The system comprises: A business layer component, used for acquiring the image; An access layer component, used to set configuration data for the image, the configuration data is used to define at least one first visual task for processing the image, any of the first visual tasks is a linear communication task including at least one subtask, each subtask points to a corresponding microservice, the microservice corresponds to three target visual processing models of a single model, a combined model or a fusion model, the configuration data defines the target visual processing model called by the microservice, specifies a score conversion table corresponding to an output label for a single model, specifies an atomic model to be used, a score table of the atomic model and a final score conversion table for a fusion model; specifies an atomic model to be used and a score table of the atomic model and a combination capability expression for a combined model; Microservice cluster, used to manage each microservice; Model layer components, used to manage various visual processing models; The access layer component is further used to determine the first subtask according to the linear communication model corresponding to the second visual task, call the microservice corresponding to the first subtask, so that the microservice corresponding to the first subtask processes the image to obtain the corresponding first processing result, and the second visual task is any of the first visual tasks; when the second visual task is completed, the output results of each subtask of the second visual task are processed based on a numerical expression to obtain the output result of the second visual task, and the numerical expression includes four arithmetic operations or a Boolean expression to express the logic of the combination of the output results of each subtask; When the first subtask is completed, a third visual task is determined, which is any first visual task that includes the first subtask and is different from the second visual task; a third subtask that is located after the first subtask in the third visual task is determined; and a microservice corresponding to the third subtask is called to obtain a processing result corresponding to the third subtask; The output results of the second visual task and the third visual task are integrated to obtain a new processing result.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement an image processing method as described in any one of claims 1 to 5.
9. An electronic device, characterized in that: It includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements an image processing method as described in any one of claims 1 to 5 by executing the instructions stored in the memory.
10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, an image processing method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Image processing engine processing method, image processing method for terminal, and terminal
CN110232652A
Image processing device
CN112001836A
Data processing method and device, electronic equipment and storage medium
CN113159091A
Service process processing method and device, medium and electronic equipment
CN113407314A
