A messaging platform system and method enriched with metadata integration

Abstract

The invention relates to a method which is integrated into the messaging platforms (B) running on user devices (A), such as phones, tablets and computers, and enables users to quickly find past multimedia messages (C) The invention provides a system that obtains metadata (D) related to multimedia messages such as photos, videos or audio recordings sent as messages, converts this metadata (D) into text and transmits it to users.

Description

[0001] DESCRIPTION

[0002] A MESSAGING PLATFORM SYSTEM AND METHOD ENRICHED WITH METADATA INTEGRATION

[0003] Technical Field:

[0004] The invention relates to a system which is integrated into social media and messaging platforms, obtains metadata related to multimedia messages that can be visual, video or audio recordings sent as messages, and converts these metadata into text and transmits them to users.

[0005] State of the Art:

[0006] Today, social media and messaging platforms are frequently used. As a messaging platform, popular applications such as Whatsapp and Telegram are known to exist. The description describes a system that can be integrated with a social media or messaging platform.

[0007] In the known art, the users can send various images, videos, audio recordings and texts to each other on messaging platforms. The users can search for their last sent messages by typing keywords into the search interface of the platform and access the messages sent or received in the past. However, this only applies to text messages. This is not the case for the messages containing images, videos and audio recordings.

[0008] In the state of the art, the users have to go through the photos, videos or audio recordings archived in the chat history of the messaging platform one by one. If the users know the sending date of the message or the subject of the multimedia message, they can access the relevant multimedia message in a short time. However, if they do not know, all multimedia messages must be analyzed one by one. This is both cumbersome and tedious, often resulting in users being unable to find the multimedia content they are looking for in past messages.

[0009] In the US patent document US2023377233, which was encountered in the literature search, a messaging system using image processing technology is described. Although it is mentioned in the relevant patent that visual analysis is performed, no study has been found to enable users to access past multimedia messages more easily. In the relevant patent, there is no study on converting the metadata detected in the multimedia message into text and transferring it between the user accounts.

[0010] In the Chinese patent document CN110414404, which was encountered in the literature search, a system is described to more easily access the registered emojis on messaging platforms. This system describes a method for finding the emoji (emoticon image) related to the chat subject by typing it. If the users have a large number of emoticon images, it may take time to find the image they are looking for. The relevant patent solves this problem. The relevant patent does not describe a system that obtains the metadata of the sent multimedia messages.

[0011] In the Chinese patent document CN108574781 , which was encountered in the literature search, a messaging platform is described. The relevant patent includes a security system that prevents image leaks when the users make calls using instant messaging clients such as WeChat and QQ. Said patent does not describe a system that creates the metadata of the sent multimedia messages.

[0012] The Chinese patent document CN107578378, which was mentioned in the literature search, describes a system that improves the resolution of the images sent on a messaging platform. On messaging platforms, when you click on an image, the image becomes larger and its resolution increases. When the image is small, the resolution is poor. The relevant patent relates to the image enhancement by image processing even when the image is small. Said patent does not describe a system that creates the metadata of the sent multimedia messages.

[0013] As a result, there is a need for a new messaging platform that overcomes the state of the art and eliminates its disadvantages.

[0014] Summary of the Invention:

[0015] The invention is a system for use on messaging platforms, which exceeds the state of the art, eliminates its disadvantages and includes additional advantages. The object of the invention is to provide a system for faster search and retrieval of multimedia messages containing past images, videos or audio recordings on messaging platforms.

[0016] The system of the invention obtains metadata by processing multimedia messages transmitted on messaging platforms. This metadata includes data such as letters and numbers on the photo or video image, information about the object obtained from the Internet with the object recognition algorithms, the sender’s location information, and the time of sending. At the same time, the messages containing video and audio records are processed with audio processing techniques and the metadata is obtained. The system automatically converts this data into text, which is saved in the messaging platform database or in an external database. In this way, when the user wants to find the relevant message, they can find it by searching with keywords.

[0017] Description of the Drawings:

[0018] The invention will be described with reference to the accompanying figures, so that the features of the invention will be more clearly understood. However, this is not intended to limit the invention to these specific embodiments. On the contrary, it is also intended to cover all alternatives, modifications and equivalents that may be included within the field of the invention defined by the appended claims. It is to be understood that the details shown are shown solely for the purpose of describing the preferred embodiments of the present invention and are presented for the purpose of providing the most useful and easily understood description of both the embodiment of the methods and the rules and conceptual features of the invention. In the drawings,

[0019] Fig. 1 is a schematic view of the system of the invention.

[0020] The figures that help understanding the present invention are numbered as indicated in the accompanying drawing and their names are given below.

[0021] Description of the Reference Numbers:

[0022] 1. Module group

[0023] 10. Image processing module 11. Audio processing module

[0024] 20. Location module

[0025] 21. Content location module

[0026] 22. Timestamp integration

[0027] 30. Internet-based search module

[0028] 40. Language module

[0029] 50. Advanced image processing module

[0030] 60. Database

[0031] 70. Secondary database

[0032] 80. Processor

[0033] 90. Icon

[0034] A. User devices

[0035] B. Messaging platform

[0036] C. Multimedia message

[0037] D. Metadata

[0038] Description of the Invention:

[0039] In this detailed description, the messaging platform (B) according to the invention is only described with the examples without any limiting effect for a better understanding of the subject.

[0040] The multimedia message (C) mentioned in the invention relates to all messages including photographs or images, videos and audio recordings. The term multimedia message (C) is defined in this way in the invention and will be used with this meaning in the remainder of the description.

[0041] Fig. 1 shows a schematic drawing of the messaging platform (B) system of the invention. The system of the invention can be integrated into any messaging platform (B). There is no limitation on this. The messaging platforms (B) are software applications that run on user devices (A) which can be phones, tablets or computers. The process executed by the system processes the images and audios in the multimedia message (C) sent on messaging platforms (B) by using audio and image processing techniques with its module group (1 ). At the end of the audio and image processing processes, the metadata (D) is obtained, and said metadata (D) is converted into text and saved in the messaging platform database (60). Metadata (D) is information that describes elements of a resource or data. In the invention, the metadata (D) can be summarized as any data / information about a multimedia message (C).

[0042] To illustrate the invention with an example in which the multimedia message (C) is an image or a photograph, let us assume that the multimedia message (C) sent is a photographic image of a medicine. In the invention, the symbols, writings and numbers on the image are first detected by processing with the image processing module (10). Assuming that “Voltaren Emulgel 1.16%” is written on the medicine, the metadata (D) in the first stage is the “Voltaren Emulgel 1.16%” data detected by processing the drug image with the image processing module (10). In the invention, this data is converted into text and saved in the platform’s database (60). In the invention, the Internet-based search module (30) accesses the metadata (D) of Internet-based product information and category. When researching about an image on the Internet, the definition information “It is used in the local treatment of musculoskeletal system diseases manifested by rheumatic diseases, soft tissue traumas such as crushes and sprains, pain, inflammation and muscle tension.” and the category “medicine” are found. The definition and category information is then converted into text format and saved in the platform database (60) as a second metadata (D). Subsequently, the location information of the sender is accessed with the location module (20). (If the messaging platform (B) allows the sharing of relevant location data) This location information is converted into text as metadata (D) and saved in the platform database (60). If the users want to search for the image in this example, i.e. the multimedia message (C), they can quickly access the relevant past message by searching by location, category, description, numbers, they remember on the image, words, or a keyword about the subject. The invention provides the users with the convenience of accessing the past multimedia messages (C).

[0043] The invention comprises a module group (1 ) which obtains the metadata (D) layers by processing the multimedia messages (C) in messaging platforms (B). Said module group (1 ) runs on a processor (80). This processor (80) can be a processor (80) on the user device (A) or a processor (80) on an external server. The metadata (D) obtained by this module group (1 ) is converted into text by the module group (1 ) and saved in the platform database (60) by a processor (80). There is more than one method in the invention to facilitate the search of the metadata (D). These are described below. In the first alternative of the invention, the metadata (D) is obtained and then converted into text and saved in the platform database (60). As a second alternative, a secondary database (70) may be used in which only the metadata (D) of the multimedia message

[0044] (C) is stored, so that the messaging platform (B), which is the main database (60) in the invention, does not occupy its own database (60) and does not occupy space.

[0045] As another alternative, on the messaging platform (B), when a multimedia message (C) is transmitted, the metadata (D) obtained with the module group (1 ) is automatically shared with the other party user as a text message. In Fig. 1 , the user device (A) on the left is the sender and the user device (A) on the right is the receiver. A multimedia message (C) sent to the messaging platform (B) on the receiver user device (A) of the user appears to contain the metadata (D) text. In yet another alternative, the metadata

[0046] (D) text can be made visible or hidden via an icon (90). When the users search with keywords to find the relevant multimedia message (C), all visible or hidden metadata (D) texts are searched and the image, video or audio recording in the relevant multimedia message (C) is retrieved.

[0047] In the invention, there is a module group (1 ) which processes the multimedia message (C) in the messaging platform (B) with image and audio processing techniques and converts all accessible data about the multimedia message (C) into the metadata (D) layers, and this module group (1 ) is operated on a processor (80). In the invention, the module group (1 ) comprises an image processing module (10). The image processing module (10) analyzes the text, numbers and symbols on the multimedia message (C), whether it is a photo or video, and converts it into text. The image processing module

[0048] (10) obtains the most basic metadata (D) about photo and video multimedia messages (C). In the image processing module (10), each frame of the videos is processed individually with image processing techniques, while the photo is analyzed through the image.

[0049] In the invention, the module group (1 ) comprises an audio processing module (11 ). If the multimedia message (C) is an audio recording or a video, the audio processing module

[0050] (11 ) uses audio processing techniques to discover frequently used keywords in the video and to determine the topic and category of the video. It then converts this metadata (D) into text. The Internet-based search module (30), which is another module used in the invention, performs search on the Internet using the first metadata (D) obtained from the multimedia message (C). It converts the metadata (D) obtained as a result of its search into text as definitions and categories.

[0051] A location module (20) in the invention converts the location information of the person sharing the multimedia message (C) and the person with whom the multimedia message (C) is shared (if the messaging platform (B) or the user has permission) into the text as metadata (D). The date of sending the message is also retrieved from the messaging platform (B), converted into the text and included in the metadata text (D). A content location module (21) in the invention is able to access location information using GPS data at the time the video was shot, if the multimedia message (C) is a video, and saves this data as the metadata (D) text.

[0052] The invention comprises a timestamp integration (22) which obtains the time at which the multimedia message (C) is sent from the messaging platform (B) and converts it into the metadata (D) text and saves the metadata (D) text obtained to the database (60) or secondary database (70) of the messaging platform (B) via the processor (80), or enables it to be transmitted to the other party as a text message in such a way that it can be made visible or hidden.

[0053] A language module (40) in the invention translates the obtained metadata (D) texts into the language preferred by the user. The language module (40) can also translate the metadata (D) text into the format required for disabled users.

[0054] The advanced image processing module (50) of the invention is a module that runs with algorithms that can not only recognize images or frames in multimedia messages (C) with photo or video content and extract text from them, but also analyze the emotional state, aesthetic value or popularity trends of the objects in the image. The advanced image processing module (50) processes the images with artificial intelligence algorithms and converts the metadata (D) into text.

[0055] In the invention, the metadata (D) is automatically detected by processing the images in photographs or videos with the image processing techniques and the audios in video and audio recordings with the audio processing techniques. Alternatively, it is possible for the users to add the metadata (D) text to the multimedia message (C) that they send as a message or to the multimedia message (C) that they receive as a message. In order to achieve this in the invention, via the icon (90) on the messaging platform (B), the user can access the metadata (D) text related to the multimedia message (C) and subsequently edit the metadata (D), e.g. by adding and removing.

Claims

CLAIMS1. A method which is integrated into the messaging platforms (B) running on user devices (A), such as phones, tablets and computers, and enables users to quickly find past multimedia messages (C), characterized in that• a module group (1 ) running on said user device (A) or a processor (80) of an external server processes a photo, video or audio recording in a multimedia message (C) transmitted on a messaging platform (B) by image processing or audio processing techniques,• said module group (1 ) obtains metadata (D) about said multimedia message (C) by means of image or audio processing techniques,• the resulting metadata (D) is converted into text format on said module group (1 ),• the metadata (D) converted into text format is saved in the database (60) of the messaging platform (B) via a processor (80) so that it can be matched with past message searches on messaging platforms (B).

2. A method according to claim 1 , characterized in that the metadata (D) converted into text format is saved in the secondary database (70), in which only the metadata (D) of the multimedia message (C) is saved, in order not to occupy extra space in the database (60) of the messaging platform (B).

3. A method according to claim 1 , characterized in that the metadata (D) converted into text format is transmitted as text messages to the messaging platforms (B) of the users.

4. A method according to claim 1 , characterized in that the metadata (D) converted into text format is transmitted as a text message to the messaging platforms (B) of the users in such a way that it can be made visible or hidden via an icon (90).

5. A method according to claim 1 , characterized in that• a multimedia message (C), which may be a video or a photograph, is processed by an image processing module (10) in the module group (1 ) using image processing techniques,• said image processing module (10) detects the text, numbers and symbols on said video or photo frames and converts these data into metadata (D) text,• the resulting metadata (D) text is saved in the database (60) or secondary database (70) of the messaging platform (B) via the processor (80), or transmitted as a text message in such a way that it can be made visible or hidden.

6. A method according to claim 1 , characterized in that• a location module (20) in the module group (1 ) converts the location information of the person sharing the multimedia message (C) and the person with whom the multimedia message (C) is shared (if the messaging platform (B) or the user has permission) into the metadata (D) text,• the resulting metadata (D) text is saved in the database (60) or secondary database (70) of the messaging platform (B) via the processor (80), or transmitted as a text message in such a way that it can be made visible or hidden.

7. A method according to claim 1 , characterized in that• an Internet-based search module (30) within module group (1) performs a search on the Internet with basic metadata (D) collected about the multimedia message (C),• metadata (D) obtained as a result of the search is recorded as text,• the resulting metadata (D) text is saved in the database (60) or secondary database (70) of the messaging platform (B) via the processor (80), or transmitted as a text message in such a way that it can be made visible or hidden.

8. A method according to claim 1 , characterized in that a language module (40) in module group (1 ) translates the resulting metadata (D) texts into the language preferred by the user.

9. A method according to claim 8, characterized in that the language module (40) translates the metadata (D) text into the format required for users with disabilities.

10. A method according to claim 1 , characterized in that• an advanced image processing module (50) in the module group (1 ), which not only recognizes the multimedia messages (C) containing photos or videos and extracts text therefrom, but also obtains metadata (D) with artificial intelligence algorithms that can analyze the emotional state, aesthetic value or popularity trends of objects in image frames,• metadata (D) is saved as text,• the resulting metadata (D) text is saved in the database (60) or secondary database (70) of the messaging platform (B) via the processor (80), or transmitted as a text message in such a way that it can be made visible or hidden.1 1 . A method according to claim 1 , characterized in that• the metadata (D) is obtained by an audio processing module (11 ) in the module group (1 ) by processing the multimedia messages (C) containing video or audio recordings by means of audio processing techniques,• said metadata (D) is saved as text,• the resulting metadata (D) text is saved in the database (60) or secondary database (70) of the messaging platform (B) via the processor (80), or transmitted as a text message in such a way that it can be made visible or hidden.

12. A method according to claim 1 , characterized in that• a content location module (21 ) operating within the module group (1) accesses location information using GPS data at the time the video was captured, if the multimedia message (C) is a video,• said location information is saved as metadata (D) text,• the resulting metadata (D) text is saved in the database (60) or secondary database (70) of the messaging platform (B) via the processor (80), or transmitted as a text message in such a way that it can be made visible or hidden.

13. A method according to claim 1 , characterized in that the user can access the metadata (D) text related to the user multimedia message (C) via an icon (90) and subsequently edit the metadata (D), e.g. by adding or removing.

14. A method according to claim 1 , characterized in that the user can access the metadata (D) text related to the user multimedia message (C) via an icon (90) and subsequently edit the metadata (D), e.g. by adding or removing.

Images